This article is a contribution by Boming Yang, who completed PFN’s 2025 summer internship program and worked part-time for the company.

日本語翻訳版: https://tech.preferred.jp/en/blog/building-a-strong-japanese-reward-model-on-plamo/

Building a Strong Japanese Reward Model on PLaMo

A reward model scores candidate responses by how well they match a human preference, and it is one of the core pieces of LLM alignment. Almost all open reward models, and almost all of the benchmarks that measure them, are built for English. For a Japanese-first setting that is a gap worth closing. A good Japanese reward model has to judge Japanese well, but the data and benchmarks to build and measure one barely exist.

This post is about how far we can get on that problem, building a strong Japanese reward model on the PLaMo architecture almost entirely from English preference data. The hard parts were not in the training. They were in measurement. We had to translate an English benchmark into Japanese to evaluate at all, and translation brought two problems with it. A translated benchmark can lose the very signal it is supposed to test. And a reward model tends to score highest on the translator it was trained on. Once we measured Japanese reward quality without that second bias, by averaging over three independent translators, our 31B model reached the Japanese accuracy of a 72B reference at less than half the size, and led every open reward model we tested. It stays competitive in English.

Background and objective

Our goal was a Japanese reward model that does not give up English, and that can be trained on data licensed for commercial use. The license constraint matters. Several of the strongest open preference datasets are research-only, so we could not use them. High-quality Japanese preference data is also scarce. That left English preference data as our main raw material, and turned the project into a translation question: can abundant English preference data be turned into a reward model that judges Japanese as well as one trained on native Japanese data?

For training data the commercial-use constraint did most of the choosing. We used NVIDIA’s Nemotron-Cascade RLHF preference data [15], which is released under CC BY 4.0 for commercial use and bundles a curated subset of well-known English preference sets (HelpSteer2 [4], HelpSteer3 [5], WildGuard [6], and others). For evaluation we used the usual reward benchmarks (RewardBench [1], M-RewardBench [3], RewardBench-2 [2]). To train, we take a PLaMo-3 [13] causal LM, replace its output head with a single scalar head, and fit it with a Bradley-Terry [10] preference objective. The English side was well covered by existing resources. The Japanese side was not, and that shaped everything after.

Building a Japanese evaluation set

Japanese reward benchmarks are scarce. M-RewardBench has a Japanese pairwise slice, where the model picks the better of two responses. But RewardBench-2 uses a harder, more discriminative format, and it had no Japanese version. Without it we could not measure Japanese reward quality on the difficult cases, which are the ones that matter most.

So we built a Japanese RewardBench-2 by translating it. This is harder than ordinary localization, because a preference dataset is not text you simply make read well in another language. Each item says that one response (chosen) is better than another (rejected), and that gap is the whole point of the item. A translation that reads beautifully but shrinks or flips the gap is useless. So the translation has to preserve faithfulness, the gap between chosen and rejected, formatting such as code and LaTeX, every field, and consistent terminology across the item.

Translating the preference data

Our first attempt translated each field on its own: the prompt, the chosen response, and the rejected response as three separate calls. Each piece came back in fluent Japanese, but the translation was quietly damaging the data. When a translator sees the rejected response by itself, it tends to clean up the very faults that make it the worse answer, which narrows the gap the item is meant to encode. It can also get the meaning wrong while reading perfectly. In one case “bastard princess” came back as わがまま (selfish) and バスター (buster): fluent, wrong, and invisible to any automated check.

Figure 1: Translate the item as a whole, not field by field. Translating the prompt, chosen, and rejected fields in separate calls leaves the translator without shared context, so one source word (“bastard”) can come back as two different, wrong readings across fields (バスター / わがまま): uncontrolled and inconsistent. A single combined call sees the whole item and renders the term one consistent way (私生児), keeping terminology aligned across the three fields.

Translating the whole item in one call fixes this. With the prompt and both responses in front of it, the translator keeps the worse answer worse and uses the same terms across all three fields. The difference is measurable. We translated the same RewardBench-2 items both ways and scored them with reward models that had been trained on neither translation. Every one of them scored higher on the combined version, by about three points on average (Figure 2). The field-by-field translation had washed out part of the chosen-versus-rejected signal, and the single-call translation kept it.

Figure 2: Translating an item all at once preserves the preference signal. The same RewardBench-2 items, translated field by field versus in one combined block, scored by reward models trained on neither translation. All five score higher on the combined-block version.

From then on we translated everything in one block, and built three Japanese versions of RewardBench-2, each with a different translator: Gemma 4 31B [12], Qwen3-235B [11], and plamo-translate[14], our own model for Japanese translation. (A fourth, from gpt-oss, was dropped after it leaked its chain-of-thought into the output.) The obvious next step was to pick the most trustworthy of the three. As it turned out, no single one could be trusted on its own.

Self-translation bias

Having three translations of the same benchmark is what exposed the second problem. When we scored our Japanese reward models, the Japanese numbers looked great, until we noticed they moved depending on which translator had produced the test set. The effect was systematic: each model scored highest on the benchmark made by the same translator it had been trained on. The Gemma-trained mix reached 85.5 on the Gemma translation but dropped to 72.1 on the plamo one. The Qwen-trained and plamo-trained mixes each peaked on their own translator in the same way (Figure 3).

Figure 3: Each reward model scores highest on its own translator’s benchmark. Three PLaMo-3 31B mixes (gemma-mix, qwen-mix, plamo-mix) on the three Japanese RewardBench-2 translations. Each one peaks on the benchmark made by the translator it was trained on.

This is the translation version of a bias already known from LLM-as-judge evaluation [7], where a model favors text written in its own style. A translator leaves a recognizable style on its output, and a reward model trained on that style rewards it again at test time. So a single translated benchmark measures translator match as much as it measures Japanese reward quality, and it cannot fairly compare models that were trained on different translations.

The cause also points to the fix. No single translation is a fair judge, but the three biases pull in different directions, so averaging across them reduces translation bias. We take the mean RewardBench-2 accuracy over the Gemma, Qwen, and plamo translations and call it rb2-jp-avg. On this metric no model gets a home-translator advantage, and the public reference models, which were trained on none of these translations, are scored on the same footing as ours. Every Japanese number below is rb2-jp-avg.

Mixing English and Japanese

With an honest metric, the training recipe became clear, and a little surprising. We compared three training sets built from the same Nemotron-Cascade pairs: English-only, the original English data; Japanese-only, its plamo-translate Japanese translation; and the mix, both concatenated so every pair appears in both languages. Training on the translated Japanese alone is a bad recipe. It gives up English: English accuracy falls from 81.0 for the English-trained model to 73.4. And measured without the translator bias, it does not actually beat the English-only model on Japanese. Its apparent Japanese gain was the self-translation bias from the previous section. It showed up only on the matching translator’s benchmark and disappeared once we averaged across translators.

What improved Japanese performance was mixing the English preference data with its Japanese translation and training on both. The mix kept almost all of the English-trained model’s English accuracy and pushed translator-neutral Japanese above the English-only baseline. It was the only setup that was strong in both languages at once (Figure 4).

Figure 4: Mixing reaches both languages. PLaMo-3 31B trained on English, Japanese, or both. English-only is balanced but not top in Japanese, Japanese-only gives up English, and the mix is highest in Japanese while holding English.

More data was not the lever. When we combined as many as seven preference sources, accuracy eventually went down. What moved the benchmarks was the proportion of each domain in the mix, not the number of sources. Reward-model quality behaved more like a data-composition problem than a data-volume one.

Results

For the final model we translated the Nemotron-Cascade data into Japanese in combined-block mode with plamo-translate[14], mixed it with the English source, and trained the 31B PLaMo on the result. We compare it on rb2-jp-avg against a proprietary reference, NVIDIA’s 72B Nemotron-Cascade reward model [15], and the open Skywork-Reward family [8, 9]: the V2 Qwen3-8B and Llama-3.1-8B models, and the earlier Gemma-2-27B and Llama-3.1-8B. Figure 5 gives the full per-benchmark breakdown.

Figure 5: Experiment results. Six reward models across the RewardBench-family benchmarks: RewardBench and M-RewardBench (v1), RewardBench-2 and RewardBench-2 JP-avg (v2). The v1 benchmarks are near-saturated; the gap that separates models is on Japanese v2, where our 31B model leads.

On translator-neutral Japanese, our 31B model reaches 82.9. That matches, and slightly edges out, the 72B Nemotron-Cascade reward model at 82.6, a gap within our measurement noise, at less than half the parameter count. It leads the strongest open model, Skywork-Reward-V2 [8, 9], by about five points (77.9). Plotting the same numbers against model size makes the efficiency clear: the 31B reaches the 72B’s Japanese with roughly 40% of the parameters (Figure 6). The same efficiency shows up in the training data. Our model was trained on about 81k preference pairs. By comparison, Skywork-Reward-V2 was curated from a pool of 40M [9], roughly three orders of magnitude more. In English our model stays competitive, 80.9 against the 72B’s 82.6, and the v1 benchmarks are near-saturated for everyone and no longer separate the models. The result we trust is the Japanese one: on a benchmark built specifically not to favor any single translator, a model less than half the size is level with the 72B and clear of every open model we tested.

Figure 6: Translator-neutral Japanese (rb2-jp-avg) against model size. The PLaMo-3 31B mix reaches the 72B reference’s Japanese at about 40% of the parameters (dashed line) and tops every open reward model, all from about 81k training pairs.

Conclusion

On top of a capable PLaMo base, building a strong Japanese reward model came down to careful data work and honest measurement. Three lessons carry beyond this project.

First, translating preference data is about keeping the signal, not just the wording. Translate the chosen and rejected responses together, or a field-by-field translation will quietly erode the gap you are trying to encode.

Second, a machine-translated benchmark carries a self-translation bias: a reward model scores its own translator too highly. Japanese reward quality cannot be read off any single translated benchmark, and has to be averaged over independent translators.

Third, training on translated Japanese alone gives up English without a real Japanese gain, while mixing it with the English source lifts Japanese. And the composition of the data, not its volume, drives the result.

The model we ended with is a PLaMo-3 31B reward model that matches, and slightly edges out, a 72B reference on translator-neutral Japanese at less than half the parameters. It leads every open reward model we tested, and stays competitive in English. Our evaluation is limited by the data and benchmarks we had, so we do not claim a universal best. But under an honest, bias-controlled measurement it is a strong Japanese reward model, and the recipe behind it (combined-block translation, a translator-neutral metric, and an English-Japanese mix) should carry over to other attempts to build Japanese reward models from English data.

References

• [1] Lambert et al. RewardBench: Evaluating Reward Models for Language Modeling. arXiv:2403.13787, 2024.
• [2] Malik et al. RewardBench 2: Advancing Reward Model Evaluation. arXiv:2506.01937, 2025.
• [3] Gureja et al. M-RewardBench: Evaluating Reward Models in Multilingual Settings. ACL 2025. arXiv:2410.15522, 2024.
• [4] Wang et al. HelpSteer2: Open-source Dataset for Training Top-performing Reward Models. arXiv:2406.08673, 2024.
• [5] Wang et al. HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages. arXiv:2505.11475, 2025.
• [6] Han et al. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs. NeurIPS 2024. arXiv:2406.18495, 2024.
• [7] Panickssery, Bowman, Feng. LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024. arXiv:2404.13076, 2024.
• [8] Liu et al. Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs. arXiv:2410.18451, 2024. Models: Skywork-Reward-Gemma-2-27B-v0.2, Skywork-Reward-Llama-3.1-8B-v0.2.
• [9] Liu et al. Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy. arXiv:2507.01352, 2025. Models: Skywork-Reward-V2-Qwen3-8B, Skywork-Reward-V2-Llama-3.1-8B.
• [10] Bradley, Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39(3/4):324–345, 1952. JSTOR 2334029.
• [11] Qwen Team. Qwen3 Technical Report. arXiv:2505.09388, 2025.
• [12] Gemma Team, Google DeepMind. Gemma 4. Model card, 2026. ai.google.dev/gemma · HF: google/gemma-4-31B-it.
• [13] Preferred Networks & NICT. PLaMo 3. Our reward models are built on an internal PLaMo-3 checkpoint, post-trained in-house from the publicly released base models plamo-3-nict-8b-base and plamo-3-nict-31b-base. 2026. HF: 8B base · HF: 31B base.
• [14] Preferred Networks. *PLaMo Translation Model (plamo-2-translate).* A dedicated English–Japanese translation model (10B, built on PLaMo-2-8b). 2025. HF: pfnet/plamo-2-translate.
• [15] Wang et al. Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models. arXiv:2512.13607, 2025. Reward model: nvidia/Qwen2.5-CascadeRL-RM-72B. Preference data (NVIDIA, CC BY 4.0, commercial-use; a curated subset of HelpSteer2/3 and WildGuard): nvidia/Nemotron-Cascade-RL-RLHF.