Nine frontier models went through Quant-Arena, a new benchmark for end-to-end quantitative research on A-share data. The best of them, GPT-6 Astra, still lost to a portfolio that just holds every stock at equal weight. That result is the most useful thing in the paper, and it says less about GPT-6 than about how we've been measuring agents.

A score is not a diagnosis

When an agent fails a task, you get one bit: fail. You don't learn whether the model lacked the capability or had it and lost it inside the harness. Tool-call formatting, retry policy, context truncation, a subagent that returned the wrong slice of data. All of it collapses into the same zero.

A paper on bridging foundational and agentic capabilities goes at that problem directly. The authors build two profiles per model: a foundational profile from standard capability benchmarks, and an operation profile from interaction trajectories. The distance between predicted and observed operation performance is the capability realization gap.

That framing earns its keep. Across fifteen model services on τ²-Bench, foundational profiles only partly predicted how models behaved at the operation level. Models with near-identical capability profiles diverged in execution, and the gaps stayed consistent across task subsets. Consistency is the part that matters. Noise averages out when you slice a benchmark. A gap that survives slicing is a property of the model plus harness.

What the five studies actually grade

The papers slice agent evaluation differently, and the units of measurement don't line up at all. Here's the map before the details.

Study	What it grades	Unit of evaluation	Holdout discipline
Bridging foundational and agentic capabilities	Where execution departs from capability	A single operation inside a trajectory	τ²-Bench task subsets, checked for consistency
Quant-Arena	The full quant research loop	One frozen strategy per model	2022–2023 visible, 2024 sealed
Environment Duels	Authoring and solving interactive tasks	One environment plus its privileged hint	Cross-model duels and RL transfer to held-out benchmarks
ToM2Vec	Partner modeling under uncertainty	A fixed budget of latent tokens per step	Partner families never seen in training
LineageEvo	Evolutionary factor search	A lineage of factor variants	CSI 500 and S&P 500, measured separately

Read the holdout column twice. Three of the five enforce a real separation between what the model sees and what it's scored on. The other two lean on cross-model transfer instead. Both choices are defensible. Neither is common in agent leaderboards.

Quant-Arena and the sealed year

Quant-Arena fixes the part of quant research that most benchmarks skip: the loop. You propose a market hypothesis, implement it as an executable signal, test it on history, decide what to keep, then propose again. Existing benchmarks score a formula, a code snippet, or one trading decision. None of them require you to close the loop.

The design puts a wall in the middle of the calendar. An agent builds features and scoring code and gets feedback only on 2022 and 2023. Then it freezes exactly one strategy. That strategy runs against 2024, which the model never saw. A cash-account evaluator applies trading constraints and costs, including market impact estimated from a simulated order book, so a strategy that churns gets charged for it. Realized excess return dominates the score, with signal quality and feature diversity as secondary terms.

Nine frontier models ran it. GPT-6 Astra scored highest. Not one beat an equal-weight A-share portfolio in 2024.

Quick Take: The current ceiling for LLM agents on end-to-end quant research is "don't lose to the dumbest possible allocation," and that's a more honest headline than any leaderboard rank.

An equal-weight portfolio is a free baseline. A model that loses to it has produced something that looks like alpha in-sample and is probably beta plus noise once the sample ends. For anyone building a research agent, the 2024 seal is the part worth copying, not the rankings.

Environment Duels: authors and solvers are different skills

Environment Duels inverts the usual benchmark. Instead of solving tasks somebody else wrote, each model writes environments. Every environment ships with a privileged hint, and the design target is a task that a capable solver fails without the hint and passes with it. That's a tight spec. The author has to know where the difficulty lives and where the escape hatch is.

Nine frontier LLMs played both roles, and the two abilities came apart. The strongest solvers were not the strongest authors. Smaller models, oddly, learned more from watching how others played, and they wrote the hardest environments and the most useful hints.

The duel structure produced something static benchmarks can't: a stable ranking. Head-to-head outcomes were transitive, and independent authors converged on similar designs. Convergence across labs is a weak signal of shared priors, and worth watching as a contamination proxy.

The reinforcement-learning result is the one I'd bookmark. Using Qwen3.8-27B, the authors trained a learner on environments from different authors. The learner's hint lift on an author's environments predicted how much training on those environments improved it on held-out reasoning and agentic benchmarks, whether the environments came from itself or another model. If that relationship holds at larger scale, environment quality becomes a measurable quantity rather than a judgment call, and the teacher model stops being a matter of taste.

Theory of mind in a fixed token budget

Multi-agent coordination needs a model of the other agent. Text is a poor medium for it. When an LLM writes out its guess about a partner's mental state, it commits to one story. Uncertainty flattens into prose.

ToM2Vec keeps the theory of mind in latent space. Each agent spends a fixed budget of continuous reasoning tokens modeling the partner. During training, those tokens are supervised against the partner's actual thoughts, so they learn to abstract and match them. At test time the tokens condition the agent's own thoughts and actions.

The ablation is the interesting part. Push the same reasoning into text and performance drops. Five environments cover collaborative, competitive, and mixed-motive settings: Overcooked-AI, multiparty negotiation, Decrypto, Hanabi, and Collab Escape/Capture. ToM2Vec beat the base model and a set of trained and prompted baselines across them, and it transferred to partner families it never trained against, including GPT-5.4.

The authors also went after the obvious alternative explanation: maybe the tokens encode task shortcuts. Their check asks the agent directly about partner beliefs, desires, and next actions. The tokens help there too, which points at a partner model rather than a policy hack.

None of these evaluations is large by ML standards. The chart below counts what each study actually ran, mixing model sweeps with environment and market counts.

Lineage control in factor search

Formulaic alpha mining has a search problem. An LLM proposes variations on a symbolic expression, fitness selects, repeat. The failure mode is concentration: the search keeps mutating a handful of structurally similar factors because they scored well early, and the population collapses inward.

LineageEvo adds a control level between the individual and the population. A lineage is the chain of variants descending from one factor. Two mechanisms use it. Lineage-conditioned variation distills search experience scoped to a lineage into operator-specific priors and fuses them with global priors, so the next mutation gets guidance relevant to that branch. Lineage-stratified allocation spreads the search budget across lineages, balancing factor quality against concentration.

On CSI 500 and S&P 500, LineageEvo had the strongest predictive performance. The allocation piece produced a more balanced lineage distribution and a structurally broader factor pool. Diversity here isn't cosmetic. A pool of factors that all fire in the same regime is one bet with extra steps.

Key Numbers

• Nine frontier models graded end-to-end in Quant-Arena, with zero beating an equal-weight A-share portfolio in the sealed 2024 window.
• 2022–2023 is the only feedback an agent gets; 2024 is sealed until the strategy is frozen.
• Fifteen model services in τ²-Bench, where foundational profiles left execution gaps that persisted across task subsets.
• Five environments in ToM2Vec, with transfer to a partner family the model never saw during training.
• Two markets, CSI 500 and S&P 500, for LineageEvo's search-control comparison.

What the community is saying

The first time I swapped a model in a production agent because it scored a couple of points higher on a public benchmark, latency went up and the failure rate went up with it. Same harness, same tools, same tasks, worse outcome. That was the day I stopped trusting cross-paper comparisons.

That experience isn't unusual. The complaints I keep running into cluster around a few things. Scores move when the harness moves, so someone else's leaderboard number tells you little about your scaffold. Trajectories are expensive, and running a task twenty times to measure variance costs more than most teams budget for evaluation. Tool schemas leak into results in ways nobody documents, and temperature zero isn't deterministic in practice.

The uncomfortable one is authorship. Teams building self-improvement loops assume the strongest model should also generate the training tasks and environments. Environment Duels says that assumption breaks. I watched a smaller model produce sharper, harder environments than a frontier model that solved the same tasks better, and the hint quality was the visible difference.

On the quant side, the recurring complaint is overfitting. Anyone who has run an agent on market data has watched it produce a beautiful equity curve that dies the moment the sample ends. A sealed evaluation year is the cheapest defense available, and almost nobody outside the benchmarks does it.

What trips people up

1. Reading an end-to-end score as a diagnosis. If a task fails, pull the operation trace before you touch the prompt. The gap sits at specific operations and it stays there across task subsets. Fix the operation, not the prompt's tone.
2. Tuning against the holdout. If your team can see the sealed window, it isn't sealed. Any signal you extract from it turns a real out-of-sample test into a slow-motion in-sample one.
3. Serializing theory of mind into prose. ToM2Vec's text ablation shows a measurable drop when partner reasoning is written out. A natural-language scratchpad is the wrong place for beliefs about other agents.
4. Optimizing factor search on best fitness alone. Track lineage concentration alongside predictive performance, or you'll end up with ten factors that are the same bet wearing different expressions.
5. Assuming the best solver is the best teacher. Pick authors and solvers with separate evaluations. The two skills came apart across nine frontier models, and picking one model for both roles costs you on one side.

One thing to remember

Every result in this cluster is a measurement of a model plus a harness, and most of the variance you'll see locally comes from the harness. Copy the evaluation discipline (sealed windows, operation-level traces, author-versus-solver splits) before you copy any leaderboard.

The bottom line

If your agent passes evals and fails in production, stop swapping models. Instrument at the operation level first. The gap usually lives in tool handling, context management, or retry behavior, and it will follow you to whatever model you try next.

If you're building an LLM research agent for markets, steal Quant-Arena's calendar rather than its rankings. Freeze one strategy, evaluate on a window nobody on the team has seen, and compare against equal weight before you believe any excess return.

Watch the latent coordination work closely. ToM2Vec's finding that text-based partner modeling underperforms continuous tokens, combined with transfer to an unseen partner family, is the kind of result that reshapes multi-agent frameworks. Expect latent coordination channels to show up as a first-class feature in agent orchestration libraries within a year.

Sources

• Bridging Foundational and Agentic Capabilities of LLMs — arXiv:2610.29751
• Quant-Arena: Benchmarking LLM Agents for Quantitative Trading Research — arXiv:2610.30113
• Environment Duels: A Microcosm of LLM Co-Evolution — arXiv:2610.30116
• ToM2Vec: Continuous Theory-of-Mind Tokens for Multi-Agent Coordination — arXiv:2610.29934
• Lineage-Aware Search Control for LLM-Based Evolutionary Factor Mining — arXiv:2610.29663