Every long-context model ships with the same invoice: attention costs N². Three papers landed this month with three ways to stop paying it, and they don't agree on anything except the target.

One rebuilds edge attention from a symmetry argument borrowed from lattice QCD and arrives at a formula that is forced rather than designed (arXiv:2610.37656). One keeps the linear-attention skeleton intact and fixes the part where it throws away variance (arXiv:2610.37645). One deletes attention entirely and puts a bounded-degree graph in its place (arXiv:2610.37524).

Read together, they map the actual shape of the problem. Two of the three are about how you compress a sequence. The third is about what attention was supposed to mean in the first place.

First, the one that sounds like a joke

The title is "Attention as Lattice Gauge Theory: Relations Interact Only Through Wilson Loops." I assumed it was one of those physics-to-ML transplants that produces a pretty invariant and no baseline. It has a baseline, and the baseline loses badly.

The setup is relational rather than token-level. Relations compose through the entity they share. Alice is Bob's mother, Bob is Carol's father, so Alice is Carol's grandmother. Order matters: a father's mother is not a mother's father. Checking a composed relation means walking a loop, Alice to Bob to Carol and back to Alice.

Now grant each entity its own coordinate frame, rotatable independently. That's the only assumption. Three consequences follow, and none of them are design choices:

• In the pure-gauge sector, relations that share no cycle cannot interact at all.
• The only gauge-invariant score of a triangle is a class function of its holonomy $H = T_{ij} T_{jk} T_{ki}$, which is a Wilson loop.
• The only equivariant update of an edge is $U = \Psi(H) T_{ij}$, with $\Psi$ conjugation-equivariant.

The punchline: edge attention mediated by a third vertex is the unique form the symmetry permits. AlphaFold's triangle update and the Edge Transformer already have that shape, an edge updated through a third vertex, but without the constraints. Commuting-operator knowledge graph embeddings turn out to be the abelian sector of the same structure. Lattice QCD has been building it equivariantly on lattices since before any of us had GPUs.

The paper instantiates the equivariant sub-family on general relational graphs as holonomy attention. No training, composition only. It infers unobserved $S_4$ relations on sparse graphs at 0.88 accuracy with a fifth of the edges corrupted. Every commutative architecture is capped at 0.083 on the same task. That is a 10x gap, and it means your knowledge graph can arrive with 20% of its edges missing and still be composed correctly with zero learned parameters in the loop.

The holonomy score also flags corrupted triangles at AUC 0.99, where commuting scores top out at 0.76 because of a second selection rule. An AUC that high means the score works as a corruption detector, not just a link predictor. You can run it over an existing graph to find bad edges without labels.

Why the symmetry argument is worth more than the accuracy

Most architecture papers hand you a point in design space and leave you to guess how big the space is. This one hands you a boundary. If you're doing edge-level attention over relational structure, the search over "how should the edge aggregate its neighbors" is closed. It's a class function of the holonomy. Anything else either breaks the frame symmetry or collapses to the abelian case.

That reframes a bunch of existing work. Triangle updates in protein structure models, edge transformers in graph learning, and the commuting embeddings in KG completion are all points on the same line, and the paper tells you which end of the line is more expressive.

Hold the enthusiasm to one constraint: this is a statement about relations, not tokens. $S_4$ is a small symmetric group. Whether a graph derived from messy text has the frame structure the argument needs is an open question, and the paper doesn't answer it.

Quick Take: Three papers, three unrelated mathematical tools (gauge symmetry, second-moment statistics, hierarchical graph topology), and the most surprising result is that the one with no training at all beats a whole family of trained embeddings by 10x on corrupted relational graphs.

MERLIN: the mean was never enough

Multi-state linear attention splits context into segments and compresses each into its own memory state. A query then reads out over those states. The compression is where the quadratic cost disappears, and the readout is where quality leaks out.

Existing MSLA aggregation uses only the mean alignment between a query and each segment's keys. A segment can have identical mean alignment to another while containing one key at very high similarity and the rest near zero. The mean cannot tell those apart, so the router sends the query to the wrong state.

MERLIN fixes this by tracking the first and second moments of each segment's keys. From those moments a query recovers both the mean and the variance of its alignment with every segment, then aggregates per-state readouts with that variance in hand.

The cost is one extra accumulator per state. That is the entire overhead. The gains on multi-state variants of Gated DeltaNet-1.3B and Mamba-2-780M: +8.3% and +7.0% on RULER needle-in-a-haystack and long-context QA, +13.5% and +12.0% across LongBench subtasks.

At 1.3B and 780M parameters you're on a single 24GB card, so these are numbers you can reproduce this afternoon. The memory comparison matters more than the percentages. MERLIN beats baselines that consume 4x more memory, which says the win comes from the routing rule and not from buying a bigger state budget.

Key numbers0.88 relation inference accuracy for holonomy attention with 20% of edges corrupted, against a 0.083 ceiling for commutative architectures. 0.99 AUC on corrupted-triangle detection, where commuting scores stop at 0.76. +8.3% and +7.0% long-context gains from MERLIN, at less memory than a baseline using 4x more. 8192x zero-shot length extrapolation for GTFormer: trained at 512 tokens, evaluated at 4,194,304.

GTFormer: throw away attention, keep the tree

GTFormer is the most aggressive of the three. It replaces flat self-attention with a syntax-free hierarchical graph neural network over the raw token stream, in the lineage of algebraic graph learning work the authors build on.

The hierarchy has five parts: a balanced binary reduction tree of depth $\lceil \log_2 N \rceil$, horizontal boundary-bridging shift edges, random expander shortcuts, multi-channel convex highway gating, and a top-down broadcast pass.

The headline result: trained exclusively on sequences of length $N \le 512$, GTFormer extrapolates zero-shot to $N = 4{,}194{,}304$ on a single commodity GPU, in linear memory, across eight long-context code, log, and multi-hop retrieval benchmarks. Standard Transformers and RoPE variants either collapse out of distribution or run out of memory.

The depth arithmetic is the whole trick. At 512 training tokens the tree is 9 levels deep. At 4,194,304 it's 22. Depth doubles while sequence length grows 8192x. Nothing in the model has to learn a new positional scheme, because there is no positional scheme to learn.

For scale: 4 million tokens is roughly 3,000 pages of text, or an entire mid-size repository, on one GPU. That's the number that makes people sit up.

The caveat is in the name. Syntax-free is a design choice, and the evaluation set is code, logs, and retrieval, domains where hierarchical structure is real. GTFormer assumes the tree is the right prior. On prose where meaning depends on long-range coreference rather than containment, that assumption is untested.

Three architectures side by side

	Holonomy attention	MERLIN	GTFormer
Core move	Edge updates only through closed loops	Track first and second key moments per state	Replace attention with a bounded-degree hierarchy
Complexity	Triangle enumeration, dominates on sparse graphs	Same as MSLA plus one accumulator per state	Linear time and memory by construction
Training needed	None for composition	Fine-tune or train as usual	Trained from scratch at 512 tokens
Headline evidence	0.88 vs 0.083 on corrupted $S_4$ graphs	+13.5% LongBench at 1.3B	8192x zero-shot extrapolation
Drop-in for an existing LLM	No, relational graphs only	Yes, if you run MSLA	No, different backbone
Biggest unknown	Scaling from $S_4$ to text-derived graphs	Whether gains survive at 70B	Behavior outside code and logs

The column that matters for planning is "drop-in." MERLIN is the only one you can bolt onto a model you already have running. The other two change what the model is.

Common pitfalls

Reading "linear" as "free." Linear attention with an undersized state budget is slower in wall-clock than a well-tuned FlashAttention kernel at 32K context. GTFormer's O(N) is a memory claim first. The tree does fewer comparisons, but the broadcast pass and the gating channels still cost real FLOPs. Benchmark at your actual sequence length, not at the asymptote.

Judging moment routing on needle-in-a-haystack alone. Single-needle retrieval is one detail in a large haystack, and it barely exercises variance. The routing fix pays off when a segment has mixed content, which shows up in the aggregation and multi-hop LongBench subtasks. If your eval suite is all needles, you'll measure +8% and conclude the idea is marginal.

Assuming a symmetry-derived architecture is automatically fast. The holonomy result constrains what the update must look like, not how many FLOPs it takes. Triangle enumeration is expensive on dense graphs, and sparse-graph bookkeeping is mandatory. Don't drop it into a dense adjacency matrix and expect the theorem to pay for the compute.

Treating 8192x as a quality claim. Zero-shot length extrapolation means the model doesn't crash and keeps near-perfect accuracy on eight specific benchmarks. It does not mean quality at 4M tokens on your corpus. Training at 512 tokens means the model never saw a document, only windows.

Confusing "no training" with "no tuning." The 0.88 figure is on $S_4$ with 20% edge corruption. Move to $S_5$, to a real schema with a hundred relation types, or to a different corruption rate, and that number will move. $\Psi$ still has to be chosen, and the choice is conjugation-equivariant but not unique.

References

1. Attention as Lattice Gauge Theory: Relations Interact Only Through Wilson Loops. arXiv:2610.37656
2. Multi-State Linear Attention with Generalized Second-Order Routing (MERLIN). arXiv:2610.37645
3. GTFormer: A Syntax-Free Graph Network Alternative to Transformers for 8192x Length Extrapolation in Linear Time. arXiv:2610.37524

One thing to remember

None of these three is going to change what you ship next quarter except MERLIN, and that's precisely because MERLIN doesn't ask you to change anything fundamental. It's a routing rule bolted onto an architecture you already run, it costs one accumulator per state, and it shows up on benchmarks you already report. The other two are more interesting and less useful today.

The Bottom Line

If you're running Mamba-2 or a Gated DeltaNet stack and your long-context numbers are flat, add second-moment routing before you buy more memory. You're currently leaving 12 to 13 points on LongBench behind a mean-alignment aggregation, and the fix costs one accumulator per segment.

If your problem is relational, entity-linked documents or a knowledge graph with missing edges, try holonomy attention first. It needs no training run and it detects corrupted edges at AUC 0.99, which makes it a cheap audit pass over data you already have. Skip it if your graph is dense, because triangle enumeration will eat the savings.

If you need million-token contexts on a single GPU, GTFormer is the only one of the three claiming it, and 8192x zero-shot extrapolation is a real result. Wait for replication outside code and log benchmarks before you bet a product on it, and watch the 70B-scale story on MERLIN, which is where its +13.5% either holds or evaporates.