Five assistants, one audit

Ten minutes after I published a write-up about AI tools, someone in the comments asked about yet another one. Not the one I had reviewed. The one I hadn't. So I stopped defending my picks and did the thing I should have done months ago: I ran the same onboarding audit across five assistants in one week.

The rules were simple. Chat only, no skills, no memories, no projects. Free tiers wherever possible, one enterprise license where not. A vanilla install for every tool, because I wanted to see what each one does when you give it nothing but a prompt.

The tools separated less on raw model quality and more on how each one handled context: what it grabbed, what it hid, what it silently forgot. Two distinct layers emerged. The conversation layer, where most things break. And a plugin layer that's quietly forming around these tools, which looks like it might fix a lot of what broke.

Claude felt like pair programming

Claude came closest to the experience I actually want. I asked it to regenerate project files with changes applied, and it did, exactly as instructed. It stuck to publicly accessible content and didn't pull context from other chats, which let me run several conversations in parallel and combine the answers. All on a free tier using Sonnet 5, and I only hit message limits twice over two weeks.

It also hallucinated, and handled the accusation well. Earlier this year I asked about a tool for measuring Slack thread response times, and Claude invented one. When I asked for sources, it said "oops, I made that up." That admission matters more than it sounds like. A model that confesses is one you can work with.

There were two moments where it held a confident position that fell apart under questioning. It cited NSA secure coding best practices for redaction workflows that made no sense on a real enterprise budget: do the redaction with one vendor, then verify with another. When pushed, it backed down. "Not really a citable best practice, but my recommendation based on security best practices." Anyone who struggles to push back on an authority figure would have accepted that. Asking "why" and "cite your sources" turned out to be the critical skill of the whole audit.

Tool	Tier	Project scaffolding	Honesty about limits	Context behavior	Verdict
Claude (Sonnet 5, chat)	Free	Full React project, clear multi-file instructions	Admitted "oops, I made that up"	Stuck to public content, chats isolated	Best pair programming feel
Gemini	Free	Partial files, assumed an existing project	Not tested	Clean but inferred what it couldn't see	Decent, not smooth
Microsoft Copilot	Enterprise	Generated a project	Not tested	Grabbed context across chats, inferred my employer from a file tree	Over-answers, weakest coding
Codex	Fresh install	Fast React PoC, stubbed the key integration	Not tested	Directory edits gated on approval	Fast, needed pushback
GitHub Copilot	Free	Project in under a minute, then broke	Confessed no web access only after 21 messages	Hid its biggest limitation until it mattered	Requirement drift

Gemini was fine at debugging. It told me exactly which errors to expect, where, and what the debugging statements should reveal. But where Claude handed me a complete React project, Gemini produced .JSX and .CSS files and expected me to drop them into a project that didn't exist. It had received the same prompt as Claude, and somehow inferred I was already working from an existing codebase.

Microsoft Copilot was the opposite problem. It grabbed internal context so aggressively that I toggled off every data source and Work IQ setting, and it still reached into other conversations. It inferred I was an Apryse employee from the directory names in an uploaded file tree. It also over-answered everything. Ask for a short answer, get a dissertation with the short answer bolded at the bottom. The coding was worse: a generated project riddled with outdated package versions, and when I brought the errors back to the chat it suffered instantaneous amnesia. "I don't have access to that." You don't have access to the code you generated one message ago?

The audit in numbers: five tools, two weeks, one enterprise license. Claude hit its message limit twice. Codex needed 9 prompts before it started guessing where its sample PDFs came from. GitHub Copilot made 3 failed attempts to fix a package version, then tried to replace the core library after 21 messages, all for 0.2 credits. Cost wasn't the problem. Behavior was.

Web access is part of the task contract

Codex felt like Claude without the orange logo. It built a React PoC that loaded quickly, and after a few rounds of back-and-forth I had a working PDF viewer and extraction logic that handled missing or redacted data. It asked before modifying files, which I appreciated. But it stubbed the WebViewer component it had recommended 30 seconds earlier. Why mock out the exact thing you're supposed to be evaluating? I suspect it was scared of the license key. I pushed, it replaced the stub, and the real viewer worked.

GitHub Copilot was the one that broke my heart. The project scaffold came in 5 seconds, a full workspace in under a minute. It. Is. So. Fast. Then the dependency errors started. First it suggested moving the WebViewer package from 9.x to 10.0.0. Then to latest, "because this is a PoC anyway." Then it announced a full switch: Apryse isn't properly installed, let's use pdf.js instead, it's simpler and already works.

I was having zero issues with the package. The assistant was guessing versions from memory, and its memory was stale. When I asked what sources it was using, it told on itself: "I'm making educated guesses about package names, API structure, configuration options. I can't verify current versions on npm. I don't have access to official docs."

Nowhere in the prior 21 messages had it mentioned that it couldn't browse the web. Every other tool I tested defaults to internet access. This one had a toggle buried in GitHub settings, and the toggle was off. I flipped it, reran the same prompt in a fresh chat, and got a working Apryse-based PoC in 10 minutes. Different project structure, zero package issues, front end and back end both wired correctly.

The lesson I took from that: treat web access as part of the task contract, not a background assumption. If a tool can't browse, it should say so up front and mark third-party integrations as unverified. Failing closed beats failing with confidence.

Quick Take: The model matters less than the harness around it, and the harness that hides its own limits is the one that burns you.

Why assistants drift when errors pile up

The Copilot failure followed a predictable mechanism. IDE agents don't just send your text to the model. Every turn injects file contents, error traces, directory listings, and tool output. A 15-message conversation is 20,000+ tokens of dense prompt context, most of it things you never typed.

The three failed package-version attempts put compiler errors in the most recent few thousand tokens. Transformers weight recent context heavily. The original requirement ("must use Apryse WebViewer") was floating 15 turns back in plain conversation history, while the immediate, undeniable goal was "make the red squiggles disappear." Switching libraries is the path of least resistance. The assistant treated its own speculative workaround as ground truth because nothing in the prompt pinned the original requirement as immutable.

The comments on this got more useful than the original post. One framing stuck with me: you don't trust your favorite coworker absolutely, you've calibrated against their failure modes. You know which one invents flags, which one apologizes and rewrites, which one is excellent for ten minutes and then quietly stops reading your message. The same calibration applies per tool and per task, and every model update shuffles the ranking. I now keep a small set of prompts I re-run after each upgrade, an onboarding audit for the tool itself. And on context boundaries: the best results come from giving an agent exactly what it needs, not everything it can access. More context made the answers worse, not better.

The plugin layer is the real news

The drift problem has a structural fix emerging, and it doesn't come from a bigger model. The fix is file-based plugins.

Anthropic open-sourced 11 knowledge-work plugins built for Claude Cowork and compatible with Claude Code. Each one is a specialist for a job function: productivity, sales, customer support, product management, marketing, legal, finance, data, enterprise search, bio-research. Each bundles the skills, slash commands, connectors, and sub-agents for that role. Install from the command line:

claude plugin marketplace add anthropics/knowledge-work-plugins
claude plugin install sales@knowledge-work-plugins

Trail of Bits went further in the security direction with a marketplace of roughly 44 skills: C and Rust security review, Semgrep rule creation, supply chain auditing, YARA authoring, constant-time analysis, Lean 4 proof writing, even a tarot-based planning skill called let-fate-decide. They already used these skills to find a real bug, a timing side-channel in ML-DSA signing, and reported it with the credit "Found using Trail of Bits Skills."

	anthropics/knowledge-work-plugins	trailofbits/skills
Focus	Job functions and company workflows	Security analysis and testing
Plugin count	11	~44
Format	Markdown + JSON, no code, no build steps	Same Claude plugin structure
Compatibility	Claude Cowork, Claude Code	Claude Code, also loads in Codex
Standout feature	Role bundles with rich connectors (Slack, Snowflake, HubSpot, Figma)	Found timing side-channel in ML-DSA

The count matters less than compatibility: Codex can load Claude plugin marketplaces directly, no sidecar metadata needed. The format is becoming a cross-tool standard, which is exactly how package managers get started.

Plugins are just markdown files

The structure is almost boring, which is the point. Every plugin is a directory with a manifest, an MCP config for tool connections, a commands folder for slash commands, and a skills folder for domain knowledge.

plugin-name/
├── .claude-plugin/plugin.json
├── .mcp.json
├── commands/
└── skills/

All of it is markdown and JSON. No code, no infrastructure, no build steps. That means skills get versioned in git, reviewed in PRs, and forked per company. "Plugins are just markdown files. Fork the repo, make your changes, and submit a PR." The vendor docs say the real value comes from customization: swap connectors in .mcp.json, drop your terminology and org structure into skill files, adjust workflows to match how your team actually operates rather than how a textbook says to.

People are already building process on top of this. The BMad Method is an open-source Agile AI-Driven Development framework that treats durable context as a first-class artifact. The pitch is direct: coding assistants are good at implementation, but they turn unstated assumptions into code. BMad responds by right-sizing the process. Clear changes go straight to build. Complex work gets deeper planning. Decisions stay explicit instead of living in chat history. It ships as a plugin marketplace for Claude Code and Codex with modules for test architecture, game dev, and a builder for making your own skills.

That's the drift fix. Requirements become files. Context carries forward across chats. The speculation still happens, but it can't quietly override something that's written down.

The fun layer matters too

Not all the tooling news is serious. The AI Avatar extension puts a VRM avatar in your VS Code sidebar that reacts to Claude Code and GitHub Copilot, or in a browser side panel for ChatGPT and Claude. Version 20 added a voxel flower garden idle mode that grows while Claude Code runs, a kaleidoscope background mode, Piper TTS as a lighter-weight option than Kokoro, and the Electron-based spin-offs: Cursor Avatar, which pops up wherever you click, and Notification Avatar.

The notification one fills a real gap. Claude Code and Codex don't have stop and approval notifications by default, and people running long agent sessions miss the moment the assistant stalls. A simple Hooks script that surfaces "the agent stopped" is the kind of tooling the vendors should have shipped themselves. With 300+ installs on the VS Code extension alone, clearly the demand is there.

My favorite detail is the 3D flower. The author wanted a simple flower in Three.js, and the AI produced a cylinder with a sphere jammed on top. It took step-by-step hand-holding: five thin oval petals as thin cylinders arranged around a center. One commenter matched the war story: four hours trying to get an AI to generate a torus knot, then gave up and wrote the vertex math by hand. LLMs are terrible at spatial reasoning, and the workaround is honest: boxes are the one shape they can reliably produce, hence the Minecraft-style voxel garden. There's a lesson in there about matching tool strengths to medium.

Common pitfalls

• Assuming web access. The Copilot failure wasn't a bad model, it was silent offline mode generating confident guesses from stale memory. Check the browsing toggle before you start any integration-heavy work, or state the contract in your first prompt.
• Leaving requirements in chat history. Error traces push old instructions out of the attention window. The fix is externalizing the spec: a pinned requirements file that gets injected every turn, or a method like BMad that carries decisions forward as files.
• Accepting the confident stub. Codex recommended Apryse WebViewer, then mocked it out 30 seconds later. Before you review generated code, ask what it stubbed and why. A PoC with the core integration mocked proves nothing.
• Trusting context isolation. Microsoft Copilot reached into other chats and inferred my employer from a directory tree. When you want an outside perspective, use a clean directory and a separate profile, and assume nothing you give it stays local.
• Skipping the update audit. Every model upgrade shuffles strengths and failure modes. Keep five prompts you re-run after each release, and treat the results as the real changelog.

One thing to remember: your favorite assistant is the one whose failure modes you've learned and whose context you've controlled. Every benchmark table fades next to that calibration, and it only comes from deliberate testing.

The bottom line

If you're building integration-heavy PoCs with third-party SDKs, verify web access and pin exact versions in an external spec file first. Otherwise you'll watch an assistant generate stale package names, then swap your core library to escape the errors it created.

If you're standardizing AI workflows for a team, adopt file-based skills instead of a shared prompt library. Markdown and JSON plugins get versioned, reviewed, and customized in git, and they survive the conversation drift that raw chat history always hits.

One thing to watch: cross-tool plugin compatibility is already here, and the marketplace is becoming the package manager for agent behavior. Teams that start writing their own skills now will have a head start when the format finishes standardizing.