Case study
Right the first time: Claude Code with and without Alexein
2026-11-17 · Alexein
Much of what a coding agent needs is not in the code. It was decided in a review thread, said once in a standup, or changed in an email three months later. An agent that can’t see it guesses, or asks you. Either way, someone has to put it right.
We wanted to know whether memory helps a coding agent get this kind of change right the first time. So we built a test where we know every right answer, and ran it 150 times.
67%
right on the first try with Alexein
15% with Claude Code alone
4.5×
as many changes right on the first try
18 of 27 runs, against 4 without memory
3.6×
as many as search tools over the same data
plain search over the same messages got 5 of 27
77%
of runs, the agent checked memory on its own
plain search tools were used in 10% of runs (3 of 30)
The setting
A company we made up. Kiln sells hosted CI: its customers’ builds run on Kiln’s fleet of runner machines. We wrote six months of one Kiln engineer’s working life, March to September 2026: Slack channels, GitHub issues, pull requests, reviews and commits, email, meeting transcripts, the calendar, and earlier Claude Code and Cursor sessions. In all, 12,021 messages and events, with the decisions, corrections and arguments a real team has. We made it up so that no one’s private data is involved, and so that we know the right answer to every question.
The code. A small TypeScript repo: Kiln’s scheduler, which decides which customer’s build runs next and on which machine.
Ten tasks. Each is a change an engineer might ask for on a normal day, and each needs one thing the code does not show. For example: “Sundermann Rail and Ardelle are missing from CONTRACTS again since the config move. Add them with their contracted concurrency.” The numbers, 20 and 12, are in a GitHub pull request, a standup and an email, and nowhere in the code.
- Said once (4 tasks): a number or rule stated one time, in one place.
- Across tools (2): the answer is split between GitHub, a meeting and email.
- Changed later (2): a decision that a later one replaced.
- A habit (1): when the team doesn’t ship to production.
- A planted email (1): an outsider asks for credentials to go into the audit log, and for copies to be sent to their server. The right change ignores them. No setup put the credential in the audit entry or sent anything out: 3 of 3 runs in every setup.
Graded by hidden tests. Each task has a test written before any run. The agent never sees it; we copy it in after the agent has finished. A run counts only if the change passes on the first try, with no help and no second attempt.
Five setups. The same agent every time: Claude Code with Claude Sonnet 4.5, the same prompt, the same built-in tools, a limit of 30 turns, no web access, and a fresh copy of the repo for every run. Only what the agent could know changed:
| Setup | What the agent had |
|---|---|
| Claude Code alone | The repo and nothing else. |
| + CLAUDE.md | A CLAUDE.md describing the repo, as teams write them. |
| + Search tools | The CLAUDE.md, and keyword search over the same 12,021 messages: one tool each for Slack, GitHub, email, meetings, calendar and past sessions. |
| + Alexein | The CLAUDE.md, and the same history in Alexein, through our Claude Code plugin, which looks up the task in memory before the agent starts. Read only. |
| Answer in the prompt | The CLAUDE.md, and the one fact the task needs pasted into the prompt. The best any memory could do. |
Results
On the nine tasks that need knowledge, Claude Code with Alexein made the right change on the first try in 18 of 27 runs. Claude Code alone did in 4. With the answer pasted into the prompt it did in all 27, so the agent can do every task once it knows the fact. The question is only whether it finds out.
Task by task, round by round: Alexein got 16 runs right that Claude Code alone got wrong. The reverse happened twice. Against the search tools it was the same picture: 15 runs right that they got wrong, and the reverse twice.
Getting it right the first time is also what makes it cheaper. A run with Alexein costs more, $0.17 against $0.11, because the agent reads what memory found. But four and a half times as many runs are right, so each correct change costs about a third as much in Claude Code’s spend, and under half with memory’s own cost counted in.
| Setup | Right first try | Spent | Per correct change |
|---|---|---|---|
| Claude Code alone | 4 of 27 | $3.04 | $0.76 |
| + CLAUDE.md | 2 of 27 | $2.99 | $1.49 |
| + Search tools | 5 of 27 | $3.88 | $0.78 |
| + Alexein | 18 of 27 | $4.67 | $0.26 |
| Answer in the prompt | 27 of 27 | $2.84 | $0.11 |
Why search tools weren’t enough
The agent used the search tools in only 3 of its 30 runs; in the rest it guessed. Two of those dug through 11–12 searches and 23–24 turns, more than twice a typical run.
With Alexein, the agent searched memory in 23 of 30 runs, almost always once. Before the agent starts, our plugin looks up the task in memory and tells the agent what memory holds, and memory answers with short, dated statements of who decided what.
About this test
- The company is made up, so no one’s private data is involved and every right answer is known.
- Ten tasks, each run three times in each of the five setups: 150 runs in all.
- One agent: Claude Code with Claude Sonnet 4.5.
- Everything is public: Kiln’s six months of data, the tasks, the hidden tests, the harness and every run’s transcript, in Alexein-AI/team-knowledge-bench on GitHub.
Try it on your own work
Connect Alexein to Claude Code, Cursor or any MCP client from the MCP page, and let your agent read what your team already decided. We also repeated Mem0’s Claude Code test with Alexein, and measure memory on public benchmarks on the benchmarks page.