Appearance
Cost-Aware Abstention: When Routing a Ticket Costs More Than Triage
A misrouted IT ticket doesn't fail loudly. It lands in the wrong queue, waits, gets bounced by someone who can't action it, and lands back in triage two hours later. Nobody logs that as an error. It shows up as a slow queue and a grumpy ops team.
The interesting thing about that failure mode is that it's priced wrong by almost every classification pipeline I've seen. We train a model, we optimize accuracy, we ship it, and we never write down what a wrong answer costs relative to a deferral. A paper on arXiv (2610.31657) does the bookkeeping properly, and the result is uncomfortable: the routing policy that minimizes cost is not the model that maximizes accuracy.
The break-even is 0.80, and most models never clear it
The authors use an illustrative cost model of (c_ok, c_triage, c_wrong) = (0, 1, 5). Correct route costs nothing. Human triage costs one unit. A wrong route costs five.
That ratio sets the entire decision rule. Deferring always costs 1. Routing costs 5 × (1 − p), where p is the probability you're correct. Set them equal: 5(1 − p) = 1, so p = 0.80. Route only when you believe you're right at least 80% of the time. A model that's correct 79% of the time is still worse than just handing the ticket to a human.
It gets sharper. At that 0.80 break-even probability, seven of the prompted configurations in the paper defer every single ticket. Claude Haiku with few-shot prompting routes 19 out of 1,000. The models are, by their own confidence estimates, nowhere near the bar.
Change the ratio and the bar moves. At 2:1 wrong-to-triage, you only need 0.50. At 10:1 you need 0.90. This is why publishing a routing threshold without publishing the cost assumption is meaningless.
Quick take: Under a 5:1 wrong-route penalty, calibrated abstention cut expected cost 38.1% and wrong routes from 15.3% to 3.5%, and it did it with TF-IDF and logistic regression, not a frontier model.
Calibrate, freeze, evaluate once
The paper frames this as a readiness assessment rather than a new abstention algorithm, and the protocol is the part you should actually copy. You pick your policy on calibration data, freeze everything, then look at held-out test data exactly once.
The freeze is the whole trick. If you sweep thresholds on the same split you report, you've fitted your threshold to noise and the number you publish is fiction. I've done this. We picked a cutoff on the test set, reported 91% precision on routing, and watched it decay for three months as the ticket mix drifted. One look, one number, then stop.
The reward for that discipline in this paper: the threshold selected on calibration (0.60) held up on test, and calibration-selected policies beat both always-route and always-defer in 14 of 16 settings when the authors varied the cost parameters after the fact.
What actually won
Four trained non-LLM classifiers, four LLMs, zero-shot and few-shot prompting. The winner was TF-IDF with logistic regression and a threshold of 0.60.
| Policy | Expected test cost | Wrong-route rate | Notes |
|---|---|---|---|
| Always route | 0.763 | 15.3% | Baselines the cost model |
| TF-IDF + LR, threshold 0.60 | 0.472 | 3.5% | Threshold picked on calibration |
| Always defer | 1.000 | 0% | Every ticket pays c_triage |
| Fine-tuned DistilBERT | n/a | n/a | 0.879 accuracy on 1k-ticket subset |
| Best prompted LLM | n/a | n/a | 0.477 accuracy on same subset |
Read the first and third rows together. Always-route costs 0.763. Always-defer costs 1.000. The "safe" option is the expensive one under this cost model, and abstention only pays off when it's selective. A policy that defers everything is just an expensive human queue with a model bolted to the front.
Key numbers
- 38.1% relative cost reduction, 95% CI [35.7%, 40.4%], from paired ratio-bootstrap
- 3.5% wrong-route rate, down from 15.3%
- 0.80 break-even confidence under (0, 1, 5) costs
- 0.472 expected cost per ticket for the frozen calibrated policy
- 7 prompted configurations that defer 100% of tickets at break-even
- 19/1,000 tickets routed by the most aggressive prompted configuration
The accuracy gap that should bother you
On a stratified 1,000-ticket test subset, the strongest prompted configuration hits 0.477 accuracy. Fine-tuned DistilBERT hits 0.879 on the same tickets. That's not a small gap. DistilBERT is a 66M-parameter model from 2019 and it doubles the accuracy of a frontier LLM in a zero/few-shot prompt.
That gap reframes the routing question. You aren't choosing between LLMs because LLMs are better on this task. You're choosing among them because someone decided prompting is the deployment path of least resistance. If you're building a ticket router today and you have 5,000 labeled tickets, fine-tune a small encoder and spend your LLM budget on the tickets the encoder is unsure about.
Calibration is not discrimination
This is where a lot of teams get fooled, and you should know about the two failures if you're copying the method.
Failure one: isotonic recalibration lowered expected calibration error and did essentially nothing for routing quality. ECE measures whether predicted probabilities match observed frequencies. It says nothing about whether the model can rank its own answers.
Failure two: for GPT-4o-mini with few-shot prompting, the model produced one single confidence value before recalibration and one single confidence value after. Correctness AUROC of 0.500. A coin flip. Every ticket got the same confidence, so no threshold could possibly separate the easy ones from the hard ones.
That's why the paper reports correctness AUROC alongside ECE, plus a tie-aware risk-coverage area. If your confidence scores tie, a tie-blind metric can hide a router that is functionally blind. Ask for the distribution of confidence values before you trust a threshold.
When prediction actually beats self-confidence
The second paper (arXiv 2610.31428) answers a question the first one raises. If you're going to build a predictor to decide whether a model will be right, how much can it possibly buy you over just asking the model?
The baseline is self-confidence: the probability the model assigns to its chosen answer. The result is a bound. Any predictor that outperforms self-confidence can be used to improve the model's loss, and its advantage transfers exactly in log loss through recalibration. Accounting for the predictor's extra parameters, that advantage is bounded by the candidate model's suboptimality plus the gain from those parameters.
Translated: routing is a bet on model weakness. Across 16 language models on four benchmarks, predictors gained more over self-confidence on more suboptimal models, and the advantage shrank as models improved through fine-tuning. Two-model cascades showed the same pattern.
The practical version. If your first model is already strong on your task, a learned router mostly adds parameters and latency for a marginal gain. If your first model is mediocre, the router has real headroom. The first paper's 0.477 accuracy is exactly the suboptimality that makes routing pay.
What trips people up
Tuning the threshold on your test split. The entire 38.1% number depends on the threshold being selected on calibration data and frozen. Sweep it on test and you're reporting the maximum of a noisy curve.
Treating ECE as a routing metric. A model can have beautiful calibration and an AUROC of 0.500. Calibration tells you if the probabilities are honest; AUROC tells you if they're useful for sorting.
Trusting a single verbalized confidence from a prompted model. One value per config, one tie for every ticket. No threshold can cut a constant.
Forgetting that the cost ratio is the decision. At (0, 1, 5) you need 0.80. Drop c_wrong to 2 and you need 0.50, and a policy that abstains aggressively becomes the wrong choice. Recompute the break-even whenever the downstream cost changes, and make sure someone actually knows what a misroute costs.
Assuming abstention is the safe default. Always-defer costs 1.000 per ticket under this model, worse than always-route at 0.763. Abstention is only cheap when it's selective.
One thing to remember
The policy is the model plus the threshold plus the cost assumption, and it's only valid for the dataset, prompt design, model endpoints, and costs it was evaluated against. The paper says this plainly. Treat it as an evaluation framework, not a deployment certificate.
The bottom line
If you're routing support tickets and you have a few thousand labeled examples, fine-tune a small encoder, calibrate it, and set the abstention threshold from your actual cost ratio. The 0.879 versus 0.477 accuracy gap says the encoder wins, and the 38.1% cost reduction says the threshold is where the money is.
If you're constrained to prompting because you have no labels, don't build the router on verbalized confidence. Seven of the prompted configs in this study route nothing at all at break-even, and the ones that do route are essentially guessing. Defer broadly and treat the model output as a suggestion for the human queue rather than a decision.
One thing to watch: the suboptimality bound from the second paper means every fine-tuning run quietly shrinks your router's value. If your cheap model improves, re-evaluate whether the cascade still earns its latency, because the gains from routing shrink as the base model gets better.