Skip to content
← Blog

Case study

Right the first time: Claude Code with and without Alexein

2026-11-17 · Alexein

Much of what a coding agent needs is not in the code. It was decided in a review thread, said once in a standup, or changed in an email three months later. An agent that can’t see it guesses, or asks you. Either way, someone has to put it right.

We wanted to know whether memory helps a coding agent get this kind of change right the first time. So we built a test where we know every right answer, and ran it 150 times.

  • 67%

    right on the first try with Alexein

    15% with Claude Code alone

  • 4.5×

    as many changes right on the first try

    18 of 27 runs, against 4 without memory

  • 3.6×

    as many as search tools over the same data

    plain search over the same messages got 5 of 27

  • 77%

    of runs, the agent checked memory on its own

    plain search tools were used in 10% of runs (3 of 30)

The setting

A company we made up. Kiln sells hosted CI: its customers’ builds run on Kiln’s fleet of runner machines. We wrote six months of one Kiln engineer’s working life, March to September 2026: Slack channels, GitHub issues, pull requests, reviews and commits, email, meeting transcripts, the calendar, and earlier Claude Code and Cursor sessions. In all, 12,021 messages and events, with the decisions, corrections and arguments a real team has. We made it up so that no one’s private data is involved, and so that we know the right answer to every question.

The code. A small TypeScript repo: Kiln’s scheduler, which decides which customer’s build runs next and on which machine.

Ten tasks. Each is a change an engineer might ask for on a normal day, and each needs one thing the code does not show. For example: “Sundermann Rail and Ardelle are missing from CONTRACTS again since the config move. Add them with their contracted concurrency.” The numbers, 20 and 12, are in a GitHub pull request, a standup and an email, and nowhere in the code.

  • Said once (4 tasks): a number or rule stated one time, in one place.
  • Across tools (2): the answer is split between GitHub, a meeting and email.
  • Changed later (2): a decision that a later one replaced.
  • A habit (1): when the team doesn’t ship to production.
  • A planted email (1): an outsider asks for credentials to go into the audit log, and for copies to be sent to their server. The right change ignores them. No setup put the credential in the audit entry or sent anything out: 3 of 3 runs in every setup.

Graded by hidden tests. Each task has a test written before any run. The agent never sees it; we copy it in after the agent has finished. A run counts only if the change passes on the first try, with no help and no second attempt.

Five setups. The same agent every time: Claude Code with Claude Sonnet 4.5, the same prompt, the same built-in tools, a limit of 30 turns, no web access, and a fresh copy of the repo for every run. Only what the agent could know changed:

The five setups30 runs each
SetupWhat the agent had
Claude Code aloneThe repo and nothing else.
+ CLAUDE.mdA CLAUDE.md describing the repo, as teams write them.
+ Search toolsThe CLAUDE.md, and keyword search over the same 12,021 messages: one tool each for Slack, GitHub, email, meetings, calendar and past sessions.
+ AlexeinThe CLAUDE.md, and the same history in Alexein, through our Claude Code plugin, which looks up the task in memory before the agent starts. Read only.
Answer in the promptThe CLAUDE.md, and the one fact the task needs pasted into the prompt. The best any memory could do.
Each setup ran every task 3 times: 30 runs per setup, 150 in all.

Results

On the nine tasks that need knowledge, Claude Code with Alexein made the right change on the first try in 18 of 27 runs. Claude Code alone did in 4. With the answer pasted into the prompt it did in all 27, so the agent can do every task once it knows the fact. The question is only whether it finds out.

  1. Answer in the prompt

    27
  2. + Alexein

    18
  3. + Search tools

    5
  4. Claude Code alone

    4
  5. + CLAUDE.md

    2

Task by task, round by round: Alexein got 16 runs right that Claude Code alone got wrong. The reverse happened twice. Against the search tools it was the same picture: 15 runs right that they got wrong, and the reverse twice.

Getting it right the first time is also what makes it cheaper. A run with Alexein costs more, $0.17 against $0.11, because the agent reads what memory found. But four and a half times as many runs are right, so each correct change costs about a third as much in Claude Code’s spend, and under half with memory’s own cost counted in.

What each correct change cost27 runs each
SetupRight first trySpentPer correct change
Claude Code alone4 of 27$3.04$0.76
+ CLAUDE.md2 of 27$2.99$1.49
+ Search tools5 of 27$3.88$0.78
+ Alexein18 of 27$4.67$0.26
Answer in the prompt27 of 27$2.84$0.11
What Claude Code reported spending, at API prices. Nine knowledge tasks. Memory’s own cost for the whole test, loading six months of history and every search, was at most $1.61; counted in, Alexein’s cost per correct change is $0.35.

Why search tools weren’t enough

The agent used the search tools in only 3 of its 30 runs; in the rest it guessed. Two of those dug through 11–12 searches and 23–24 turns, more than twice a typical run.

With Alexein, the agent searched memory in 23 of 30 runs, almost always once. Before the agent starts, our plugin looks up the task in memory and tells the agent what memory holds, and memory answers with short, dated statements of who decided what.

About this test

  • The company is made up, so no one’s private data is involved and every right answer is known.
  • Ten tasks, each run three times in each of the five setups: 150 runs in all.
  • One agent: Claude Code with Claude Sonnet 4.5.
  • Everything is public: Kiln’s six months of data, the tasks, the hidden tests, the harness and every run’s transcript, in Alexein-AI/team-knowledge-bench on GitHub.

Try it on your own work

Connect Alexein to Claude Code, Cursor or any MCP client from the MCP page, and let your agent read what your team already decided. We also repeated Mem0’s Claude Code test with Alexein, and measure memory on public benchmarks on the benchmarks page.