Compilation is a floor, not a ceiling

SolBehave ran seven open-weight models against 250 real, reproducible Ethereum contracts. Compilation success ranged from 26.4% to 92.7% depending on the model. Then the authors took every function body that compiled and fuzzed it against the original. Only 31.4% came back equivalent.

So the best models in that batch produced code that compiles roughly nine times out of ten and behaves correctly roughly three times out of ten. On Ethereum, deployed contracts can't be patched, which turns one silent divergence into permanent money loss.

That gap shows up everywhere in this cluster. A unit test asks whether a program passes. A compiler asks whether it parses. An official judge gives a verdict under a submission budget and explains nothing. All four papers here attack the same hole from different directions: they replace one coarse reward with several narrow, checkable signals.

• PreciseCoder swaps "did the tests pass" for "was the edit small and semantically verified."
• ARBITER swaps the judge's verdict for a counterexample you can run.
• SolBehave swaps compilation for fuzzing, historical replay, and DCR-guided call sequences.
• AesCode swaps one aesthetic score for parsable property checks plus a VLM judge bound to a design graph.

Key Numbers SolBehave: 26.4% to 92.7% compile success across seven models, and only 31.4% of compiled function bodies match the original under differential fuzzing. ARBITER: verdict accuracy 71.41% to 87.41% after RL, counterexample success 40.61% to 62.67%. PreciseCoder: average edit lines fall from 7 to 15 down to 3 at a matched pass rate. AesCode: 31.1 and 22.4 percentage-point visual-quality gains for the 8B and 32B versions.

PreciseCoder: the reward teaches the model to edit

Here's the finding that should worry anyone post-training a debugger with unit tests. Reinforcement learning from test outcomes raises pass rates and also teaches the model to touch more lines. Give a 4B model a correct program and a unit-test-only reward, and it will invent a bug and "fix" it.

PreciseCoder adds a second term to the reward: a signal for edits that are both semantically verified and localized. On 4B, 9B, and 27B models, average edit length drops from 7 to 15 lines down to 3, with pass rates unchanged. Three lines versus twelve is the difference between a patch a reviewer can read in ten seconds and one they skim and approve.

The generalization is the part I didn't expect. Trained only on single-line bugs, the model handles repository-level repair and resolves more issues with smaller patches. Then the precision transfers upward. Hand the reasoning trace of a 9B PreciseCoder model to GPT-5.6 Sol or Qwen3.8-Max and edit precision climbs 4.6 to 12.4 points with no additional training. A smaller model's thinking trace acts as a scaffold for a larger one.

The authors checked the traces linguistically. The reward pushes the model to reason about individual lines, name the buggy ones, and hold one consistent bug hypothesis instead of guessing repeatedly.

Quick Take: Every one of these papers gets its gains by measuring more things separately and keeping each measurement checkable, whether that's a line-level edit, a runnable counterexample, or a replay against a call graph.

ARBITER: counterexamples as receipts

Competitive programming judges hand down verdicts and nothing else. They don't show you the failing input. They don't tell you which branch was wrong. And each submission costs you from a finite budget.

ARBITER reframes the job. Given only the problem statement and a candidate's source, a 35B mixture-of-experts verifier judges the program and generates a counterexample input that exposes the failure. That input is evidence. Anyone can run it and see the divergence for themselves.

Training happens in a generic execution sandbox with RL. Counterexample success on genuinely incorrect programs moves from 40.61% to 62.67%. Mean binary verdict accuracy on USACO-JUDGE, built from the USACO 2026 contests, moves from 71.41% to 87.41%. That second number is the one that matters for test-time scaling. A verifier at 71% will pick the wrong candidate often enough to waste the compute you spent generating candidates in the first place.

The refinement loop is where the payoff lands. In an IOI 2026 case study, ARBITER-guided revision pushed GPT-OSS-120B's expected score above the gold-medal threshold while staying inside the official submission limit. The verifier never spent a submission to learn what was wrong.

SolBehave: replay beats a test suite

Smart contract evaluation has leaned on "does it compile" for years, mostly because nothing better was cheap to run. SolBehave applies three checks instead of one.

Differential fuzzing compares each generated function against the original on automatically generated inputs. Historical replay runs the whole contract against the transactions it actually received on-chain. DCR-guided replay, the paper's core contribution, walks sequences of function calls that follow the contract's intended usage, including orderings that never happened on-chain. A Dynamic Condition Response graph enumerates those legitimate orderings, so you're testing the call patterns users would hit, not just the ones the chain happened to record.

The results across seven state-of-the-art open-weight models, after one round of automated repair, are grim. Between 13% and 31% of compiled contracts diverge under historical replay. Between 13% and 44% diverge under DCR-guided replay. Only Gemma-4-31B stayed below 3% on both.

Read the two rightmost bars together and the practical lesson is clear. A contract that survives historical replay can still break under a call ordering that never occurred on-chain. If your test suite is a recorded transaction log, you're testing the easy half.

AesCode: rewards for things you can't parse

Generating slides, posters, and dashboards as code gives you structured, editable, verifiable output. It also asks the model to plan layout and color on a canvas it cannot see.

AesCode sidesteps that by representing the target as a design graph that describes the whole canvas and makes every property attributable to a specific element. From that graph it derives two kinds of reward. Deterministic verifiers score properties you can parse out of the code and its rendering, like element positions and color values. A VLM judge scores the rest against a sample-specific Visual Graph Rubric tied to graph elements and relations, so the judge is grading named properties instead of vibing on a screenshot.

The two reward channels get normalized separately during RL. That detail matters: without per-channel normalization, one channel's scale swamps the other and you optimize the easy one.

Against same-scale Qwen3-VL baselines given identical reference images, AesCode-8B gains 31.1 percentage points in visual quality and AesCode-32B gains 22.4, landing in the range of closed-source frontier models like GPT-5.5 and Claude Opus 4.8. Ablations show the reference image pays off twice, as model input and as a source of visual rewards.

The benchmark itself is worth borrowing. It scores verifiable correctness and visual quality on separate axes, and its visual scores track human judgments closely. If you've ever watched a code model produce a beautiful dashboard with the wrong numbers in it, you know why one combined score hides the failure you care about.

How the four signals fit together

System	What it grades	Signal it swaps in	Headline number
PreciseCoder	Debug patches	Localized, semantically verified edits	Edit lines 7-15 to 3 at matched pass rate
ARBITER	Competition submissions	Generated counterexample plus verdict	Verdict accuracy 71.41% to 87.41%
SolBehave	Solidity contracts	Fuzzing, historical replay, DCR replay	26.4% to 92.7% compile, 31.4% equivalent
AesCode	Visual artifacts as code	Parsable verifiers plus graph-bound VLM judge	+31.1 and +22.4 visual-quality points

The pattern underneath all four: take one reward that conflates several properties, split it into channels, and make each channel produce something a human or a script can independently check. PreciseCoder splits pass/fail from edit precision. ARBITER splits verdict from evidence. SolBehave splits compilation from behavior. AesCode splits parsable properties from judged ones.

What the community is saying

When I compared notes with people running these pipelines in production, three complaints came up repeatedly. The first was that their verifier and their generator shared too many failure modes. If both run on the same base model, a subtle misunderstanding of the problem statement propagates through both, and the verifier cheerfully approves garbage. ARBITER's counterexample requirement is the fix people reach for, since a counterexample that actually runs is hard to fake.

The second was diff bloat. One engineer told me their automated PR bot averaged around forty changed lines on bugs that turned out to be one-liners, and reviewers had started rubber-stamping anything under a hundred lines. That matches exactly what PreciseCoder reports about unit-test-only rewards.

The third was contract testing in a CI loop. Teams that generate Solidity wanted a cheap equivalence gate they could run on every commit, and DCR-guided replay is the first thing I've seen that's practical for that, because the call graph is derived from the contract's own interface rather than from a hand-written scenario list.

What trips people up

Wiring a judge verdict straight into your training signal before the judge can produce evidence. ARBITER's verdict accuracy only reached 87.41% once the model had to generate counterexamples. A judge that agrees with a wrong program because the program "looks right" will teach your model to write code that looks right.

Penalizing diff length without a verification term. PreciseCoder's small patches come from a signal that checks the edit semantically. If you only reward brevity, the model learns to make tiny non-fixes and your pass rate quietly drops.

Assuming your model reproduces the SolBehave spread. Gemma-4-31B held under 3% divergence on both replay regimes while other models hit 44%. That's a per-model property, not a property of the technique. Re-run the equivalence suite on every model swap, not once at evaluation time.

Pointing a VLM judge at a full rendering with a global aesthetic prompt. AesCode ties rubric items to specific graph elements and relations. Global scores drift between runs and can't tell you which property regressed.

Reusing any reasoning trace across model families. The 4.6 to 12.4 point precision gain came from a 9B PreciseCoder trace feeding frontier models. A trace from a weaker model with a different failure profile can mislead a stronger one, and you won't notice without an ablation.

The Bottom Line

Ship a counterexample generator before you ship a code judge. ARBITER's verdict accuracy only climbed to 87.41% once the model had to produce failing inputs, and a wrong verdict with no evidence costs more than no verdict at all, because you'll trust it.

Add a localized-edit reward when your training loop already passes the test suite. PreciseCoder cut patches from 7 to 15 lines down to 3 without losing pass rate, and that precision transfers from a 9B model's traces to frontier models for 4.6 to 12.4 points of edit precision with zero extra training.

Treat compilation as a gate and never as a score. On Ethereum contracts, 92.7% compile success coexisted with 44% divergence under DCR-guided replay, so version-pin an equivalence suite and rerun it on every model swap. Watch the contract-repair side over the next two quarters; Gemma-4-31B's sub-3% divergence is the number everyone else now has to beat.