Three Ways to Get Memory Out of the Context Window

Every RAG deployment I've worked on gets the same security review. Someone asks about prompt injection. Someone asks about hallucination rates. Nobody asks what happens if an attacker gets read access to the vector index.

That's the gap the first of these three papers goes after, and it sets the tone for the other two. All three treat the context window as a cache and everything interesting as happening in the memory hierarchy around it. One secures the datastore, one restructures the corpus, one trains the model that does the writing.

The datastore is the part nobody audits

Embeddings are not opaque. Inversion attacks recover a meaningful fraction of the source text from a dense vector, and published attacks have gotten good enough that "it's just floats" stopped being a defense a few years ago. If your index is multi-tenant, or shared across teams, or backed up to a bucket with looser permissions than the primary database, the vectors are the documents.

The fix in Private Retrieval Augmented Generation via Random Projection is almost embarrassingly small. Pick a random matrix R of shape d × k where k is much smaller than d. Multiply every datastore embedding by it once at index time. Multiply every query embedding by the same matrix at search time. Cosine similarity in the projected space tracks cosine similarity in the original space closely enough that retrieval quality barely moves.

Projecting both sides with the same matrix

The Johnson-Lindenstrauss lemma is why this works at all. A random projection into k dimensions preserves pairwise distances among n points to within a factor of 1 ± ε as long as k scales like log(n)/ε². For a million chunks you don't need many dimensions. For ten million you still don't.

The arithmetic is the part that sells it internally. A 1536-dimension float32 embedding is 6 KB. At ten million chunks that index is roughly 61 GB. Project to 64 dimensions and each vector is 1 KB, so the same index is roughly 10 GB. That's a 6x reduction in storage and a matching reduction in the number of bytes an attacker has to exfiltrate before they have anything worth attacking.

Key numbers

• 1536 → 64 dimensions: 6 KB → 1 KB per stored vector
• 10M chunks: ~61 GB index → ~10 GB
• Reported retrieval and generation quality beats formal-DP baselines, with comparable empirical privacy under strong attack models
• Librarian Agents: +33 points on BrowseComp-Plus, +31 points on BRIGHT zero-shot
• Hy-Memory-30B-A3B: 55.49% macro average vs 54.62% for Claude Opus 5

Quick Take: All three papers are solving the same problem from different ends, which is that a context window is a cache and the interesting engineering lives in the memory hierarchy around it.

Where the security model actually rests

The security model rests on R staying secret. Anyone who has the matrix can invert the projection and recover something close to the original embedding. Random projection is a keyed transform, and its guarantees are conditional on key secrecy in a way that differential privacy's are not.

That distinction matters if you work anywhere near a compliance process. DP gives you a worst-case bound on what any adversary can learn, and you pay for it in recall. Random projection gives you an empirical barrier that held against the attacks the authors ran. The paper's phrase is "comparable empirical privacy under strong attack models," which is carefully worded and means exactly what it says: we ran the attacks we know about, and the projected index held up about as well as the DP version.

For most teams that's the right trade. DP's recall hit is real and measurable on day one. The attack you're defending against is hypothetical and probably not targeted at you specifically. Take the cheap barrier.

Reactive retrieval hits a ceiling

Standard RAG does a top-k lookup at query time. That works when the answer lives in one or two chunks. It falls apart when the question needs you to reconcile a contract clause against an amendment against a three-week email thread. The retriever returns five chunks that each look relevant and none of them fit together.

Precomputed structures like knowledge graphs try to fix this and mostly don't. The extraction step runs on fixed heuristics, so the graph ends up shaped by the extractor's assumptions rather than by what questions actually need.

Librarian Agents takes a different angle. The starting observation is that Claude Code and similar agent harnesses are genuinely good at navigating a filesystem to solve the task in front of them. They are not trained to build a filesystem that someone else will navigate later.

So a builder agent reads the raw corpus and produces a hierarchical directory of synthetic files and reference links. The output is a workspace: a hierarchical filesystem of synthetic files and links, not a summary or an index. A separate answering agent then navigates that workspace at query time.

The formulation is a Markov decision process. State is the partially built workspace. Actions are file creation, linking, and reorganization. Reward is the downstream answering accuracy of the answering agent that receives the finished workspace. Train the builder against that reward with RL and it learns to anticipate what the answering agent will need.

The numbers and what they cost

Thirty-three points on BrowseComp-Plus is not a tuning gain, it's a different regime. Thirty-one points on BRIGHT, zero-shot to unseen domains, says the builder learned something general about how to organize a corpus rather than memorizing one benchmark's structure.

Both numbers carry the same caveat. They're measured with a specific answering agent, and the builder was trained against that agent's behavior. Swap the answering model and you retrain the builder. The 33 points are not a property of the corpus, they're a property of the corpus-builder-answerer triple.

That's also where the cost sits. Workspace construction is upfront and expensive, and it only pays off if you amortize it across repeated queries against the same corpus. One-shot questions don't benefit.

Paper	What it fixes	Mechanism	What it costs
Private RAG via Random Projection	Datastore leakage	One shared random matrix applied to both sides	Key management; a linear transform, not a formal DP bound
Librarian Agents	Reactive retrieval	RL-trained builder that reorganizes the corpus into a filesystem	Upfront construction, amortized over queries
Hy-Memory-30B-A3B	Write-path quality	On-policy distillation plus RL on a compact MoE writer	Serving a 3B-active model alongside your app

What trips people up

Treating random projection as a compliance checkbox. It's a keyed transform. If R leaks, the privacy is gone, and R usually ends up in the same config file as the indexer settings, committed to the same repo. Put it in a secret manager or don't bother.

Projecting only the datastore. Both sides need the same matrix. Project queries with a different one, or forget to project them at all, and similarity collapses to noise. This is the single most common implementation error and it fails silently, because you get results back, they're just wrong.

Picking k from the storage chart. 64 dimensions is fine for short chunks and degrades faster on long chunks and numeric lookups where fine-grained distinctions matter. Sweep k against your own eval set before you commit.

Rebuilding the workspace per query. The entire point is amortization. If the builder runs on every request you've built a very expensive retriever with a 33-point ceiling it will never reach.

Training the writer against the wrong harness. Hy-Memory's own results make this point. A writer specialized for one memory protocol does not transfer directly to another. The On-Policy Distillation stage exists precisely because the write protocol is framework-specific and the model has to learn each one.

A 30B writer that does one job

Mem0, MemOS, and Hy-Memory each have their own write protocol, and each is usually staffed by a general-purpose LLM behind a framework-specific prompt. That raises serving cost, and the prompts become the actual product, which is a bad place to be.

Hy-Memory-30B-A3B is a mixture-of-experts model with 30B total parameters and 3B active. Serving cost tracks the 3B path, so you're paying for a small model that was trained to do one thing. Two stages got it there.

Memory On-Policy Distillation teaches the framework-specific write protocols on student-generated states, so the training distribution matches what the model will actually see at inference rather than what a teacher model would produce. Memory RL then optimizes for downstream answerability with dense per-chunk probes, attributed cross-chunk rewards, over-storage controls, and chunk-level GRPO. Carrying policy-written memory banks across windows keeps training aligned with deployment.

The over-storage control is the detail I'd steal even if I never trained a writer. A model that dumps everything into memory scores beautifully on recall and destroys the reader's context budget. Penalizing storage explicitly is how you get a writer that decides what to drop.

The headline result: first place on all four evaluated harnesses, including held-out EverMemOS, with a 55.49% macro average against 54.62% for Claude Opus 5, measured on fixed 20% benchmark subsamples. The margin is under a point. Read it as parity at a fraction of the serving cost, not as a clear win. Parity at 3B active parameters is the interesting part.

One thing to remember

Every one of these systems has a component that is trained or tuned against a specific downstream consumer. The random projection is tuned to your embedding model. The librarian is trained against your answering agent. The writer is trained against your memory framework. Swap the consumer and the tuning is void, which means none of these are drop-in upgrades no matter how small the code change looks.

The Bottom Line

If you run a shared or multi-tenant vector index and can't justify DP's recall hit, add the random projection. It's a few lines of code and a key you have to actually protect. The storage reduction is a bonus, not the reason to do it.

If your users ask cross-document questions against a stable corpus, the librarian approach is where the big number lives. Budget for the upfront build and for retraining whenever you change the answering model. One-shot question workloads should skip it entirely and stay on plain top-k retrieval.

Watch the write path. A 3B-active model matching Opus 5 on memory writes means memory management stops being prompt engineering and becomes a model you deploy, version, and evaluate like any other. That shift is happening now, and it changes what your inference budget looks like.