When Preference Signals Conflict: Ties, Pareto Dominance, and Stable Alignment

Your DPO model learned to write longer answers. It didn't learn that longer is better. It learned that longer wins, because that's what your labels said often enough to matter.

That's the first failure mode: a preference pair only says "A beats B". It never says why. Any feature that happens to differ between A and B and correlates with the label gets gradient, and the model happily turns length, bullet points, or a confident tone into a proxy for quality. You didn't ask for that. The loss gave it to you anyway.

The second failure mode shows up when you have more than one reward model. Helpfulness, safety, conciseness, factuality: they disagree on the same response. Scalarize them with fixed weights and the disagreement becomes a confident-looking gradient pointing somewhere none of the reward models endorse. The usual fix is to filter those rollouts out. The usual result is that you throw away a third of your batch and call it stability.

Two recent papers go after these problems with the same underlying move: stop treating preference data as a flat list of labels, and start asking which comparisons actually carry signal.

Ties are a correction signal, not a data augmentation trick

A tie is a comparison labeled "equally good". In the DPO objective it sets the target margin to zero. The gradient pulls the implicit rewards of the two responses together instead of apart.

That's a different operation from adding more pairs. A pair tells the model which direction to move. A tie tells it which differences don't matter, and it does that by damping the gradient component along whatever features separate the two tied responses. If your tied pair differs mainly in length, the tie applies a brake to the length direction. If the two responses differ only in phrasing that nobody cares about, the tie teaches nothing and costs you an annotation.

So ties are not a free lunch, and the paper's framing is the right one: ties differ in their corrective effect, and selecting which ones to add matters as much as adding them.

Annotators produce these judgments anyway. Most pipelines record "about the same" as a discard and move on. That's a wasted label, and it's the only label in your dataset that says anything about irrelevance.

Picking ties is a subset-selection problem

For log-linear policies the authors derive each tie's influence on spurious reliance in closed form. Then they prove the obvious greedy thing: take the k ties with the largest predicted reductions, and you get the first-order optimal subset among all subsets of size k. Not a heuristic. The best you can do at that budget, to first order.

The catch sits in the middle of that diagram. Computing influence exactly needs two things you don't have: knowledge of which features are spurious, and the curvature of the training objective.

You rarely know the first. Annotators don't tag "this preference was driven by verbosity". You suspect it, you don't measure it. The second is a Hessian. Nobody is building a Hessian for a 7B model.

Quick Take: tie augmentation is a data-selection problem, and the selection signal is cheap to approximate but expensive to compute exactly.

The proxy that makes tie selection affordable

The paper explores the magnitude of a tie's implicit reward margin as a stand-in for influence. The intuition holds up: a tie where the model already assigns near-identical rewards carries little correction, while a tie where the model confidently separates two responses labeled equal is pushing hard against a belief the model holds.

That proxy isn't free either. The authors characterize when the margin tracks influence, and the conditions matter. It works roughly when the spurious direction dominates the margin, so the margin is measuring the thing you want to suppress rather than genuine quality differences, and when curvature is close to isotropic across the pair. Early in training, curvature is large and uneven, and the ranking degrades. Ranking by margin then picks ties that are expensive to fit rather than ties that are corrective.

For neural models they go further: a counterfactual proxy for spurious reliance (perturb the suspected spurious attribute, measure how far the preference logit moves) combined with a generalized Gauss-Newton approximation to curvature. For language models that GGN object is diagonal or low-rank, which means you can estimate it with a handful of Jacobian-vector products instead of a second training run.

The empirical claim across log-linear policies, small neural models, and LLMs is consistent: comparable reductions in spurious reliance with fewer ties than uniform selection. Fewer ties is an annotation-budget statement, and that's the part practitioners should care about.

The numbers that decide your tie budget

Two papers, two signals. Selective tie learning works on the ambiguity you already collect. GPDPO works on the disagreements you already compute.

3 policy classes in the tie-selection experiments: log-linear, neural, and LLM. The selection rule transfers, which is the point of testing all three.

0 extra rollouts for GPDPO. The auxiliary preference pairs come from responses already sampled and already scored.

1 auxiliary loss added to the policy objective. It's optimized jointly, not as a separate stage.

G(G-1)/2 candidate pairs per sampled group of G responses. Group structure is the resource both methods spend.

When rewards disagree, dominance is the only trustworthy signal

Multi-reward RL has a well-known failure: conflicting signals produce ambiguous update directions. The existing responses are reward reweighting, gradient coordination (project conflicting gradients onto each other), and rollout filtering.

Filtering is the crudest and the most widely used. It's also the one that throws information away. The conflict is a property of the aggregate, usually the scalarized group reward or the group-relative advantage. It is not a property of every pair inside the group. If response A beats response B on all four reward models, that's a fact, regardless of what the weighted sum says about the group as a whole.

GPDPO builds on that observation. Sample a group, score every response with every reward model, find the Pareto-dominance relations, and convert each one into an auxiliary preference pair. Jointly optimize that pairwise loss with the original group-relative objective. No new sampling. No new reward evaluation. The conflict-filtered rollout gets reused instead of discarded.

On API-Bank, across model scales and reward configurations, GPDPO beats the multi-reward baselines it's compared against. The follow-up analysis is the more interesting result: it reduces both group-relative conflicts and pairwise conflicts, which means the auxiliary signal isn't just adding data, it's changing the shape of the update.

There's a limit worth naming. Dominance pairs are sparse when rewards are anti-correlated. Safety and helpfulness trade off on a large fraction of prompts, so on those prompts you get few or no dominance relations and the method falls back toward the baseline. The gains concentrate where the reward models mostly agree but the scalarization is unstable, which is the common case in practice.

Two methods, one underlying move

Dimension	Selective tie learning	GPDPO
Broken signal	Spurious features correlated with the preference label	Conflicting rewards producing ambiguous group updates
What gets added	A selected subset of tie comparisons	Auxiliary preference pairs from Pareto dominance
Where it comes from	Ambiguity your annotators already produced	Rollouts you already sampled and already scored
Selection rule	Top-k by predicted influence on spurious reliance	Dominance relation on the raw reward vector
Extra compute	Curvature estimate, a few Jacobian-vector products	Pairwise comparisons within existing groups
Main risk	Margin proxy breaks when curvature is uneven	Dominance is sparse under anti-correlated rewards
Best fit	Single reward, noisy labels, obvious shortcut features	Multiple reward models, filtering-heavy pipelines

Both papers are doing the same thing at different layers. One mines structure from the ambiguity in a single preference signal. The other mines structure from the partial order inside a group when several signals collide. Neither asks for more data.

How many usable pairs a group actually gives you

Eight rollouts give you 28 candidate pairs. If a fifth of them turn out to be dominance relations, that's five or six free preference comparisons per prompt, from responses you were going to discard. Sixteen rollouts give you 120 candidates, which is why group size is the cheapest lever here. The cost scales linearly with G and the pair count scales quadratically.

That's also the reason GRPO-style training became standard so quickly. Group structure is a free supervision multiplier, and both of these papers are spending it more carefully than the baselines.

Common pitfalls

• Uniform tie augmentation. If you add ties without ranking them, you need far more of them to hit the same reduction in spurious reliance. You're paying annotation cost for comparisons that carry no corrective signal. Select by influence or by a proxy you've validated, and never by convenience.

• Reading a tie as a no-op. The tie loss still moves weights. It pulls the two implicit rewards together, which will collapse a real quality distinction if the tie label is wrong. Tie quality has to be at least as high as pair quality, and "equally bad" is not the same judgment as "equally good".

• Estimating influence with the reference model. Curvature is a property of the current parameters, not the frozen reference. Rankings computed once at step zero decay as training proceeds. Recompute every few hundred steps, or freeze a validated subset early and stop.

• Scalarizing before checking dominance. A weighted sum turns an incomparable pair into a confident-looking preference. Compute dominance on the raw reward vectors first, then scalarize whatever's left. Reversing the order destroys the signal you're trying to recover.

• Filtering conflict rollouts and moving on. Those groups contain the only unambiguous supervision in the batch. Filtering also biases your effective training set toward prompts where the reward models happen to agree, which usually means the easy prompts, which is exactly where you need the least help.

The one habit that transfers

Before touching a loss function, count how many of your preference comparisons are actually informative. Both papers are, underneath the math, an argument that most of your preference data does redundant work while a small selected subset does the real alignment. Ties tell you which differences are irrelevant. Pareto dominance tells you which differences are undeniable. Neither signal costs you a new annotation campaign or a new sampling pass, and both are sitting in data you already generated.

What this means for your next alignment run

• If your model is picking up length or formatting shortcuts, don't retrain from scratch and don't buy a reward model. Collect ties, rank them by implicit reward margin, take the top slice, and retrain. The margin proxy works well enough when you're already mid-training and the curvature has settled.

• If you're running three or more reward models and filtering on disagreement, stop discarding. Compute Pareto dominance inside each group before you scalarize, add the surviving pairs as an auxiliary pairwise loss, and watch your effective batch size recover. It costs zero extra sampling and zero extra reward evaluations.

• If your rewards are strongly anti-correlated, expect dominance to be sparse and don't expect GPDPO to rescue you. Go back to reward design or per-prompt weighting instead, and watch group size as the lever that matters most over the next few months.