Appearance
The Scoring Function Is the Benchmark
Georgian counts in twenties. 99 is ოთხმოცდაცხრამეტი, four twenties and nineteen. The language has its own alphabet, no grammatical gender, and no relatives in the major language families, which makes it a useful stress test for any claim about multilingual coverage.
George Kobaidze built one by hand: 287 scored items across five tasks, typed out one at a time, then reviewed by a language specialist who sent back three or four corrections. No machine translation anywhere in the set.
A hand-written benchmark, run on $10 a day
The five tasks are picked to hit different failure modes, not different topics.
| Task | Written | Kept | Scoring |
|---|---|---|---|
| Numbers | 186 rows | 82 rows (158 questions) | exact match, pipe-separated alternates accepted |
| Verbs | 115 items | 50 | multiple choice or exact match |
| Translation (ka to en) | 47 paragraphs | 12 | judge model, 0 to 4 meaning scale |
| Cloze | 2 articles | 2 articles (50 gaps) | per-gap exact match |
| Correction | 51 paragraphs | 17 | word-level error delta |
The numbers task is the one I'd steal. Georgian ordinals keep a prefix when the word starts with მე- (მე-18) and take a suffix when it doesn't (21-ე). Two rules, one conditional. A model that has absorbed Georgian text gets this right without being told. A model that pattern-matched a handful of examples gets it wrong roughly half the time, and the exact-match scorer gives it zero credit for being close.
The set shrank before the real run. Kaggle hands out $10 a day of model credit with no top-up button, so a planned 41-model, nine-dimension sweep became 8 models across 5 tasks. The trimming was done with a probe model: run a cheap model over the full set, keep every item it got wrong, drop part of what it found easy. Full sets live in the repo's archive/ folder for anyone running outside the platform.
Two findings from the run are worth generalizing. Understanding Georgian is largely solved; producing it isn't. Reading comprehension on a language with a small web footprint is a retrieval problem, and modern tokenizers plus enough pretraining data handle it. Generating correct morphology is a different skill, and the exact-match scores fall apart there.
The other: Georgian costs Anthropic's models more tokens than English for the same content. Tokenizer coverage varies by language, and it lands on your bill. If a Georgian-speaking user costs 1.6x what an English-speaking user costs on the same product, that's a pricing decision, not a curiosity.
The scoring function decides what your benchmark can see
Most teams inherit a scorer and never look at it. The correction task has the best one I've seen this year:
python
score = max(0, (errors_before - errors_after) / errors_before)Three properties fall out of that line. Partial credit is proportional. Fix three of four planted errors and you get 0.75. Over-correction is punished, because turning a correct word into a wrong one adds an error and drags the score down. And rewriting the whole paragraph in the model's own style doesn't pay, since every gratuitous change is a new difference from the reference text. That last property is the one most graders get wrong. A model that paraphrases confidently shouldn't outscore one that makes a surgical edit.
Translation goes the other direction and uses a judge model, Gemini 3 Flash, deliberately not one of the eight models under test. Five levels.
| Score | Meaning |
|---|---|
| 4 | same meaning, nothing missing, wrong or added |
| 3 | one small slip (tense, minor detail), main meaning intact |
| 2 | one important part wrong or missing |
| 1 | only fragments right |
| 0 | wrong, unrelated, or not English |
The rubric carries one line that looks like a footnote and isn't: the judge is told that he, she, and they are all acceptable renderings when the Georgian source doesn't reveal gender. Georgian has no grammatical gender, English forces a choice, and without that instruction the judge penalizes correct work. Every judge-based eval has a line like this somewhere, and if nobody wrote it down, your judge has an opinion you didn't authorize.
Key numbers from this run 287 scored items per model. Eight models, four levels. Five tasks, down from a planned nine. $10 per day of Kaggle credit, no top-up option. 32,000 max completion tokens, against a longest observed answer of 26,600. A 5% error threshold that voids the entire run. Reasoning disabled for every model, which cut the cost of a full pass by roughly 99%.
Quick take: a benchmark is the sum of its scorer, its item-writing process, and its harness, and the harness is where the number usually gets corrupted.
When the harness lies to you
I ran the same tasks through Kaggle Benchmarks and hit three problems in the first day, all of which produce a plausible-looking wrong number.
Kaggle reserves the worst-case cost of every call before making it: max output length times price. The reservation for my first top-tier run came back as max estimated cost of operation ($1.60195) exceeds your available quota, with plenty of real credit left. The actual answer would have cost a fraction of a cent. I set the output limit to 32,000 after measuring the longest real answer at 26,600 tokens, and dropped to two parallel calls.
Failed calls count as wrong answers. A run that dies halfway through reports 0.3 and puts it on the leaderboard as if the model were bad at the language. I added a guard to every task: isolated failures after three retries count against the score, but more than 5% failed calls raise and report nothing. That guard fired immediately when the reservation problem killed my first attempt, and both runs stopped cleanly instead of publishing a fake result.
Some models think before answering, and you pay for every hidden token. One test run spent roughly 770,000 hidden tokens to answer 251 questions, most of them one word long. GPT-5.5 burned about 750 hidden tokens to pick a single letter. That also makes the comparison unfair, since comparing a thinking model to a non-thinking one measures the setting more than the model. Turning reasoning off for everyone fixed both problems at once.
None of this is Kaggle-specific. Every hosted eval platform reserves cost optimistically, counts errors lazily, and lets model defaults leak into your leaderboard.
Datasets are the other half of the eval
An eval number is downstream of a dataset decision, and the current releases show how different those decisions can be.
| Dataset | Scale | What it's for |
|---|---|---|
| EgoStandard | 80,000 hours planned, head view, hands only | egocentric video pretraining with 3D hand pose |
| EgoStandard-body | 10,000 hours planned | adds full-body pose |
| EgoProStandard / -body | 8,000 / 2,000 hours planned | wrist plus hand pose, body on the body SKU |
| EgoDemo | 50 hours, stored in-repo | entry point, LeRobot v3 and MCAP plus raw MP4 |
| tiktok-5.6B-videos | 5.6B videos | content and engagement metadata for social video |
| jabarti-llm-dataset | Arabic QA pairs | difficulty (easy/medium/hard) by type (factual/temporal/analytical) |
| openai-math (FineEnvs) | Lean 4 challenges from openai/math | kernel-checked proof grading |
| FDE-Gym | 95 tasks, 20 public | agent delivery inside a customer environment |
Eighty thousand hours is more than nine years of continuous footage, so nobody is training on the full line in one pass. EgoDemo's 50 hours is a weekend of download time and enough to validate a data loader, which is the right place to start. The pose contract is specific: hands at (21,3) positions and (21,4) quaternions, full body at (22,3) and (22,4), float32 world coordinates, quaternion order (qw, qx, qy, qz). LeRobot and MCAP are two representations of the same episodes joined by source_episode_uuid, so you don't add the counts together.
The TikTok dataset is the opposite shape. Five point six billion videos with caption, hashtags, on-screen text, sound ID, is_ad, branded_content, is_ai_generated, and eight engagement counters per row. The early rows are mostly empty. A 2014 video in the sample has no hashtags and no mentions, because those fields didn't exist yet, and stats_updated_at runs into 2026 because engagement keeps accruing. Sampled engagement rates span 0.0046 to 0.327, a 71x spread. If you're training on engagement as a label, that timestamp column tells you the label is still moving.
jabarti-llm-dataset labels every Arabic QA pair with difficulty and question type. That's a small schema decision with a large payoff: it makes slice reporting possible. A model that can pull a date out of a biography and a model that can synthesize across a career both show up as correct unless you separate factual from analytical items. Tagging the split is the difference between one score and a diagnosis.
openai-math takes the grader question furthest. Each task ships with a Lean 4 statement, a Mathlib environment, a Dockerfile, a reference solution, and grading through a Comparator sandbox. A proof typechecks or it doesn't. No judge model, no rubric drift, no alignment between annotators. If you can express your task as a typecheck, you've eliminated an entire class of eval noise.
Agent benchmarks answer a different question
FDE-Gym tests whether an agent can do the work of a forward-deployed engineer: enter a customer's running environment, figure out what matters there, and ship something that holds up. Twenty scenarios are public and 75 variants are held out on the leaderboard, scored as normalized business uplift from 0 (the system as found) to 1 (the reference solution), minus stakeholder contact cost.
That score design is the interesting part. Asking a customer a question has a price. An agent that escalates every ambiguity loses points, which matches how delivery work is actually judged. Scenarios span a retail bank's anti-scam hold queue, Medicare prior authorization, a hospital triage chat, and a power company's nightly interval-settlement job. You can run one locally with python3 harness/run.py new --engagement campus_booking --level L3, and the levels run from L0 (the brief says the most) to L3 (the brief says the least).
The license forbids training on the files and carries a canary GUID, which is the right default for a benchmark whose variants are the whole point. Contamination destroys a held-out set faster than any other mistake.
Four controls that make a decision defensible
Audit has a working definition of a defensible answer: an experienced auditor, new to the work, can follow it. Translate that to an agent and you get four checks, each attached to a specific failure.
| Control | Failure it prevents | What it looks like |
|---|---|---|
| Authorization | Agent rewrites a row a human already signed | Check object state, not just caller identity |
| Execution control | Logs show the job ran but not why the result changed | Per-step inputs, outputs, and model version under a stable ID |
| Evidence provenance | Citation points at "January bank statement" | Bind to document ID, version, and hash |
| Evaluation integrity | Gold labels come from last year's conclusions | Tag each label with methodology and period; report by slice |
The authorization case is the one that should worry anyone shipping an agent into a regulated process. An auditor flags a $62,500 check to a payee missing from the vendor master, a reviewer signs off, new statements arrive a few days later, and the agent rewrites the row to "no exception." Every write was permitted for the agent's identity. The signature now vouches for an answer nobody reviewed. Scoping permissions per tool doesn't catch this. The check has to be on the object, in this state, right now.
Execution control is the reason step-level records matter. An agent says a $96,400 check cleared. A rerun reads $69,400. Job-level logs tell you the job ran, they don't let you reexamine one decision. Break the run into fetch, extract, match, conclude, store each step's inputs and outputs, and rerun the contested step in isolation.
Common pitfalls
Trimming an eval set with a single probe model. Keep every item the probe missed and drop the easy ones, and your benchmark now measures that model's blind spots by construction. The Georgian set did this, but if you copy the trick, run at least two probes and keep the union of their misses.
Reporting a score from a run that lost calls. A quota failure counts as a wrong answer, so a dead run publishes 0.3 for a model that's actually fine. Add the error guard and fail loudly instead.
Putting thinking and non-thinking models on one leaderboard. GPT-5.5 spent roughly 750 hidden tokens on a one-letter multiple-choice question. Compare it against a model answering directly and you've measured the setting.
Treating a document's name as its identity. "January bank statement" can be three different files with the same title, and the same file can change between runs. Bind citations to ID, version, and hash.
Letting an agent write in place to a signed row. An overwrite doesn't weaken the signature, it turns the signature into a claim about an answer nobody reviewed. Require an explicit re-open with its own audit event.
What the community is saying
Two threads shaped how I think about this. The comments under Fieldguide's agent-audit post on dev.to are the best short account I've read of permissions failing in practice; several people there had scoped agent access per tool and never per object state, which is exactly the gap that lets an agent overwrite a signed row. The pattern I've settled on after reading that thread is to treat a signed object as an immutability boundary, so any write touching it needs a re-open event rather than a rewrite.
One check I've added to my own review list: 96,400 and 69,400 differ by a digit transposition, and the gap, 27,000, is divisible by 9. That's the classic signature of a transposed pair, which points at the extract step rather than the model's reasoning. Divisibility by 9 on a rerun disagreement is a cheap automatic flag before anyone gets pulled in, and you can run it before you have any gold label at all.
The third thing that came out of the discussion, and the one I keep coming back to: the useful artifact for a regulated agent is the replayable basis, meaning the source identifiers, the transformations, and the rubric in force at the time. A reviewer who can trace one exception from bank evidence through to the conclusion doesn't need to reconstruct the whole run.
One thing to remember
A benchmark number is a measurement of three things at once: the model, the items, and the harness that ran them. Change the output cap and the number moves. Let one model think by default and the number moves. Count a dead API call as wrong and the number moves. When you cite a score, you're claiming all three were under control, and the teams that say so explicitly are the ones whose numbers survive a challenge.
The bottom line
If you're building a multilingual eval, don't scale the item count. The Georgian set kept 287 items and produced five findings. Spend the budget on a specialist review pass instead. Three or four corrections on hand-written items buys you more than a thousand machine-translated rows.
If your agent can write to any row a human has already approved, fix that before anything else. Every write in the $62,500 example was permitted for the agent's identity. Permission per tool is not permission per object state, and the check belongs on the object, in its current state, at write time.
If your domain has a formal checker, use it as the grader. Lean 4 plus Mathlib gives openai-math a grader that can't drift and can't be argued with. If you can express your task as a typecheck, do it, because you get a benchmark with zero judge noise and no annotator alignment problem to manage.