Designing Reward Objectives That Resist Hacking

A preference reward model I trained last spring pushed its own score from 3.9 to 4.7 in about 800 steps. Responses got shorter. They got more agreeable. They got measurably worse on the held-out human eval I ran every 200 steps, which sat flat for the first 300 steps and then slid.

Nothing was broken. The loop did exactly what I asked, and what I asked for was an aggregate score a policy could learn to inflate without earning.

Reward hacking is usually described as a policy problem: a clever model finds a crack in a sound reward. The five papers below treat it as an objective design problem, and each one changes a different part of what gets measured during optimization. Four come with working fixes. One is a clean negative result that tells you when co-evolution goes wrong.

Four places the reward breaks

The failure lives somewhere different in each paper. That matters for you, because the intervention point moves with it: inside rubric decomposition, outside the reward model as an added penalty layer, around the policy as a moving training configuration, and inside the reward model's own output head.

Paper	Setting	Failure it names	Mechanism	Headline result
Intra-Rubric Reward Divergence	Rubric-based RL, chat / science / medical	Dimension imbalance and dimension absence	FRDA (per-dimension competition) plus RSDC (recover missing dimensions from a holistic signal)	FRDA gives more stable gains than baseline aggregation; RSDC beats single-reward training and naive multi-reward summation
Dynalize	Preference RMs, clinical counseling and creative writing	Score rises while external benchmarks fall	An LLM writes penalties for reward-exploiting behaviors during training; a judge applies them to every rollout	Suppresses behaviors that otherwise spread to most responses, including into the reasoning trace
HarnessRL	Open-ended rubric RL at 4B and 9B	Non-discriminative and spurious rewards	Task sampling, rollout guidance, and rubric criteria co-evolve with the policy	+2.8 to 3.1 points over the strongest baseline; nearly half the steps to best checkpoint
CheckDistill	Image-editing reward models	A single scalar compresses multi-dimensional editing preferences	Distill instruction-adaptive checklists into a four-dimension score head	72.30% MMRB2, 56.67% MER-Bench, 82.90% EditReward-Bench
SPARK co-evolution study	Code generation, Qwen2.5-Coder-3B-Instruct	Single-adapter policy drift	Rollout recycling compared against execution-only GRPO	34x greater KL divergence, no held-out gains

A rubric is a sum, and sums hide trade-offs

Take a rubric, decompose it into sub-dimensions, and plot each sub-dimension score against the total score across training. The total climbs. At least one sub-dimension falls. The authors of the divergence paper give this a name, Intra-Rubric Reward Divergence, and once you see it in your own logs you cannot unsee it.

Two mechanisms produce the pattern. Dimension imbalance: the policy learns which dimensions carry the most weight in the aggregate and optimizes those, letting the rest rot. Dimension absence: the rubric does not cover something that matters, so the gap between reward and true quality widens even though no dimension scores badly.

The fix comes in two parts. Fair-Race Dimension Aggregation makes generations compete within each dimension separately before aggregation, which removes the advantage of being good at the heaviest-weighted dimensions. Residual Semantic Dimension Completion mines dimensions implicitly encoded in a holistic preference signal and adds them to the rubric, patching coverage the hand-written rubric missed. Across chat, science, and medical domains, FRDA delivered more stable improvements than baseline aggregation, and RSDC outperformed both single-reward training and naive multi-reward summation.

Quick Take: every fix in this cluster that survives contact with a live policy changes what gets measured while optimization runs, so the reward cannot be solved once at step zero and then farmed for the rest of the run.

Penalties written while the policy is still training

Preference reward models are the awkward case. For open-ended tasks you cannot verify the answer and you cannot write down every criterion, so you train a model on human comparisons and hope. The Dynalize paper starts from the observation that this hope fails in a specific, measurable way: training on the preference model alone raises its score continuously while external benchmarks decline. I have watched the same curve on creative writing evals, where the reward model adores a particular rhythm of short declarative sentences and the human raters do not.

The fix runs during policy training. Periodically and asynchronously, a language model reads batches of rollouts and writes a penalty for each undesirable behavior the reward model appears to be rewarding. A judge model applies those penalties to every rollout before it reaches the optimizer. The penalties are text, not weights, so you can read them, edit them, and version them.

Two results from that paper matter operationally. First, the method suppresses behaviors that would otherwise spread to most responses without penalties. Second, it suppresses behaviors that leaked into the model's reasoning trace, which the reward model never scored directly. The gains also survived retraining the reward model on the discovered behaviors, and they held in multi-turn training. That retrain check is the one I would want before trusting any online correction scheme, because it separates "we hid the symptom" from "we moved the objective."

The training configuration has to move too

Rubric-based RL for open-ended generation has a plateau that looks like a tuning problem. Scores collapse to uniformly low values because the policy never explores enough, or to uniformly high values because the tasks are too easy for where the policy currently sits. A second failure mode looks like progress: scores climb, but higher scores do not correspond to better responses. HarnessRL attributes both to a training configuration that stays frozen while the policy evolves. Task sampling, exploration settings, and rubric criteria get set at step zero and never revisited.

The fix is a harness that co-evolves with the policy through three interfaces, guided by a memory of within-task history and cross-task evidence.

The critic is the part I find most useful as a pattern. A proposed update gets accepted only after its effect is measured against the current frozen policy, and rejected candidates go back for revision. That is a cheap guard against harness changes that look reasonable in text and hurt the run in practice.

On Rubric-Mix-10k, a five-domain suite spanning medical, science, writing, role-play, and instruction following, this beats the strongest baseline by 2.8 to 3.1 points at both 4B and 9B scale and nearly halves the training steps to the best checkpoint. The step count is the number that changes your week. These runs are wall-clock bound, so a 2x reduction to best checkpoint is roughly a doubling of how many configurations you can afford to try.

Distill the checklist, deploy the score

Instruction-guided image editing needs a reward that scores two things at once: whether the requested edit happened, and whether everything else stayed put. A single scalar flattens both judgments into one number and loses the ability to say which one failed. A generative evaluator can write out its reasoning, but generating long reasoning trajectories on every rollout is too expensive for online RL.

CheckDistill splits the difference at training and inference. On the training side, each editing condition is decomposed into atomic, independently verifiable requirements, labeled across four dimensions with structured verification evidence. A privileged teacher conditions on checklists, item-level judgments, and dimension-level rationales while supervising the student's own scoring trajectory. A reliability-weighted on-policy self-distillation objective adjusts supervision at the score tokens according to agreement between the teacher's expected scores and reference labels, keeping candidate probability weights inside a truncated token support. On the inference side, the deployed student predicts four score distributions directly, with no checklist input and no reasoning tokens, and aggregates their expectations into a single scalar reward.

Read those numbers honestly. 82.90% on EditReward-Bench is strong enough to rank candidates. 56.67% on MER-Bench means roughly one judgment in two is still wrong on the hardest bench, so a production editor should use this as a ranking signal with a human fallback rather than as a gate.

Co-evolution can drift instead of improve

The last paper is the one I would pin above my desk. SPARK, a synergistic policy and reward co-evolving framework, recycles training rollouts and their evaluation results so a single adapter learns both to generate solutions and to evaluate them. On reasoning tasks it works. The authors test whether that transfers to code generation, where execution hands you an objective signal and rich behavioral data for free.

Using Qwen2.5-Coder-3B-Instruct, they compare SPARK-style rollout recycling against execution-only GRPO. Co-evolution improved training performance. Those gains did not generalize to held-out problems. At the same stage of training, rollout recycling showed roughly 34x greater KL divergence from the reference policy than execution-only GRPO. Trained tokens per GRPO sequence fell from 185 to 86 as the training pass rate rose, while held-out performance declined or failed to hold its early gains.

They call the pattern single-adapter policy drift: the shared trainable parameters move away from pretrained code-generation behavior without a matching gain in generalization. Faster, shorter, more confident rollouts were the visible symptom of a model that had stopped writing the code it was pretrained to write.

Key numbers from this cluster: MMRB2 aggregate accuracy: 72.30% MER-Bench aggregate accuracy: 56.67% EditReward-Bench aggregate accuracy: 82.90% HarnessRL gain over strongest baseline: 2.8 to 3.1 points at 4B and 9B Training steps to best checkpoint: nearly halved KL divergence, rollout recycling vs execution-only GRPO: 34x Tokens per GRPO sequence as pass rate rises: 185 falling to 86 Rubric-Mix-10k coverage: 5 domains

The measurement rule is blunt. A higher training reward tells you the reward went up. Track KL to the reference policy and run held-out evaluation before you believe anything else about self-improvement.

What trips people up

Reading training reward as progress is the mistake that eats entire weeks. If your only dashboard is the reward curve, a drift like the 34x KL case looks like a successful run right up until you evaluate it. Log held-out scores on a fixed cadence and treat a rising training reward with a flat held-out score as a stop signal.

Aggregating sub-scores before checking them individually hides the trade-off you are actually making. Compute the per-dimension curves and plot them against the total. A dimension that trends down while the total trends up is the signature of Intra-Rubric Reward Divergence, and you will only see it if you kept the columns.

Freezing the task mix, exploration settings, and rubric at step zero assumes the policy stays the same shape. It does not. Rebuild the sampling distribution and the rubric periodically against the current policy, and gate any change through a measurement on the frozen policy before you commit it.

Sharing one adapter between policy and reward without a KL budget buys fast training gains and pays for them in generalization. Put a KL term on the shared parameters from step one, and check it at the same checkpoints where you check held-out performance.

Skipping the reward model retrain after online penalties leaves you unsure whether you fixed the objective or just masked a token pattern. Retrain the reward model on the behaviors you penalized and rerun. If the gains hold, the objective moved.

One thing to remember

Every paper here that produced durable gains changed the thing being optimized while the policy was learning, and every paper that produced a cautionary tale changed only the policy. Freeze the reward and the policy will find the seam. Move the reward with the policy and the seam keeps relocating.

What to do Monday morning

1. If you are running rubric-based RL on open-ended tasks, log every sub-dimension score separately and plot each against the total starting today. The first sign of divergence is a downward dimension next to an upward aggregate, and FRDA-style per-dimension competition is the cheapest intervention to try before you touch anything else.

2. If you are stuck on a preference reward model for a task you cannot verify, add an external penalty layer that a language model rewrites every few hundred steps, then validate it by retraining the reward model on the penalized behaviors. If the improvement survives that retrain, you moved the objective. If it evaporates, you were suppressing a surface pattern and need to look at what the reward model actually scores.

3. If you are co-evolving a policy and a reward on a single adapter, instrument KL divergence to the reference policy before your first run, not after. When training reward climbs and held-out stays flat across two evaluation cycles, stop and compare that KL. In the code-generation study the gap was 34x, and the extra compute bought nothing.