Skip to content

Benchmarks

Remembers more.Reads less.

Seven public benchmarks, set beside the best published results: where Alexein leads, where it’s level, and where it isn’t first yet.

Accuracy on four benchmarks

LoCoMo

Same answer model

1,540 questions

Accuracy in percent, highest first:

  • Alexein94.6
  • MindMemOS94.03
  • Mnemis93.9
  • EverOS93.05

LongMemEval-S

500 questions

Accuracy in percent, highest first:

  • Mem0 Platform93.4
  • Alexein91.6
  • Mem0 OSSsame setup91.0

Other vendors self-report higher still.

LoCoMo-Plus

vs paper’s Table 1

401 questions

Accuracy in percent, highest first:

  • Alexein55.4
  • A-Mem17.20
  • Mem015.80
  • SeCom14.90

3.2× the paper’s best memory

MuSiQue

1,000 questions

F1, highest first:

  • HippoRAG 2F1 48.6
  • AlexeinEM 33.9F1 44.0

4.6 F1 behind HippoRAG 2.

  • One dot per question, lit up to Alexein’s score
  • Best comparable published result
  • Read clockwise from 12 o’clock; a full circle is 100

Alexein’s results are self-reported. Published results as their sources report them.

  • +58.0

    points, same model, better memory

    GPT-4.1-mini when facts keep changing: 36.0 to 94.0

  • 3.2×

    the LoCoMo-Plus paper’s best memory

    55.4 against A-Mem’s 17.20

  • #1

    memory framework on Microsoft’s RHELM

    ahead of memU, Mem0 and MemGPT; top 4 of 18 overall

  • 70.57%

    pass rate on BEAM’s million-token conversations

    level with Mem0’s 70.14%, in Mem0’s own harness

01Five pairs

Same model. Better memory.

GPT-4.1-mini answers the same questions twice: once reading the whole history, once reading the brief Alexein hands it. Only the memory changes, and in every pair the brief wins, from a fraction of the reading.

Scores out of 100.

  1. MemoryAgentBench

    Facts that keep changing

    +58.0 points with Alexein

    Without Alexein · reads 262K tokens

    36.0

    With Alexein

    94.0

    Fact consolidation: 262K tokens of facts, overwritten as they go; answer with the latest. Accuracy (SubEM), no judge. MemoryAgentBench, arXiv 2507.05257 (opens in a new tab)

  2. RHELM

    A year of someone’s chats, emails and documents

    +14.6 points with Alexein

    Without Alexein · reads the whole history, in a 1M-token window

    33.9

    With Alexein

    48.5

    All 1,305 questions, with emails and documents in memory. RHELM’s own reader and judge prompts, GPT-4.1-mini judging. RHELM leaderboard, github.com/microsoft/RHELM (opens in a new tab)

  3. LoCoMo

    Long conversations, over weeks1

    +14.0 points with Alexein

    Without Alexein · reads 26,031 tokens

    80.6

    With Alexein

    94.6

    All 1,540 questions over multi-session chats. LLM judge. Mnemis, arXiv 2602.15313 (opens in a new tab)

  4. MemoryAgentBench

    One fact, found in a long history

    +9.0 points with Alexein

    Without Alexein · reads the whole context

    83.0

    With Alexein

    92.0

    RULER-QA1, accurate retrieval. Accuracy (SubEM), no judge. MemoryAgentBench, arXiv 2507.05257 (opens in a new tab)

  5. MemoryAgentBench

    A whole detective novel, then the questions

    +9.9 points with Alexein

    Without Alexein · reads the whole book

    56.3

    With Alexein

    66.2

    DetectiveQA, all 10 books (71 questions). Accuracy. MemoryAgentBench, arXiv 2507.05257 (opens in a new tab)

  1. 1.The whole-conversation score is the Mnemis paper’s, judged by GPT-4.1-mini; Alexein’s is judged by GPT-4o-mini, majority of three.

Alexein’s results are self-reported; the whole-history scores are as their sources publish them.

On this page

02LoCoMo

At the top, by under a point.

Long conversations over many sessions, then questions about what was said. Every system here answers with GPT-4.1-mini, so the memory is what’s compared. Alexein’s 94.6 is ahead of MindMemOS, Mnemis and EverOS, but only 0.57 points above the first of them; judges vary by source and the answer key has known errors, so read the top of this board as close.

  • Alexein
  • Memory systems
  • No memory system: the whole conversation, or plain retrieval
  1. Alexein

    94.6
  2. MindMemOS

    94.03
  3. Mnemis

    93.9
  4. EverOS (EverMemOS)

    93.05
  5. EMem-G

    85.3
  6. Zep

    85.2
  7. MemOS

    80.8
  8. The whole conversation, no memory

    80.6
  9. Nemori

    79.5
  10. RAG

    73.8
  11. LangMem

    73.4
  12. memU

    66.7
  13. Mem0

    66.3
  14. PREMem

    65.8

LLM-judge accuracy. Alexein’s is self-reported; the rest as MindMemOS, arXiv 2608.12428 (opens in a new tab) and Mnemis, arXiv 2602.15313 (opens in a new tab) publish them. Judges follow each source; Alexein’s answers are judged by GPT-4o-mini, majority of three.

LoCoMo’s answer key has known errors, listed in an audit of it (opens in a new tab); they affect every top system’s score, not only Alexein’s.

On this page

03LoCoMo-Plus

3.2× the best memory in its paper.

Its questions hinge on something said long before, and never said in so many words. The paper’s memory systems answer with GPT-4o; the models with no memory read the whole conversation.

3.2×

55.4 against A-Mem’s 17.20, the best memory system in the LoCoMo-Plus paper.

  • Alexein
  • Memory systems in the paper
  • No memory system: a model reading the whole conversation, or retrieval
  1. Alexein

    55.4
  2. Gemini 2.5 Pro, whole conversation

    26.06
  3. Gemini 2.5 Flash, whole conversation

    24.67
  4. GPT-4o, whole conversation

    21.05
  5. GPT-4.1, whole conversation

    18.63
  6. A-MemGPT-4o

    17.20
  7. Mem0GPT-4o

    15.80
  8. RAGGPT-4o

    15.55
  9. SeComGPT-4o

    14.90
  10. gpt-5-nano, whole conversation

    14.84

Binary-judge accuracy. Alexein’s is self-reported; the rest are the LoCoMo-Plus paper’s Table 1 (opens in a new tab), and the comparison is to that table. A later paper reports a higher score from its own runs.

On this page

04RHELM

The top memory framework on Microsoft’s RHELM.

A year in ten people’s lives: their chats with an assistant, their emails and documents, and 1,305 questions about all of it. With the same reader as the paper’s memory frameworks, GPT-4.1-mini, Alexein ranks above every one of them, on both of the leaderboard’s tables.

Without external sources: memory holds the conversations only.

Without external sources: Alexein 35.6, 4th of 18. memU, the best other memory framework, 23.1. GPT-5.5 reading a 1M-token context, the best overall, 57.0.

Chats only, as ranked8 of the leaderboard’s 18 rows

  1. GPT-5.51M-token context57.0
  2. Claude Opus 4.61M-token context49.7
  3. Claude Opus 4.5retrieval (k=20)36.2
  4. Alexein4th of 18memory framework35.6
  5. Qwen3.5-397B-A17B1M-token context34.6
  6. memUmemory framework23.1
  7. Mem0memory framework21.1
  8. MemGPTmemory framework13.9
Chats only
+12.5
Chats, emails and documents
+14.9
Points over memU, the best other memory framework, with the same reader.

The highest-ranked memory framework on both tables, ahead of memU, Mem0 and MemGPT, the paper’s memory frameworks, each reading with GPT-4.1-mini.

Above it: frontier models reading the whole history in a 1M-token window, and on the chats-only table one retrieval setup, 36.2 against 35.6, within the ±2.6 margin.

Averages as published on the RHELM leaderboard (opens in a new tab) (paper: arXiv 2605.31086 (opens in a new tab)); Alexein’s is self-reported. The margin on an average is ±2.6 at 95%.

On this page

05MemoryAgentBench

When facts change, it keeps up.

A long run of facts, each one overwritten as it goes, then questions that ask for the latest. Reading all of it, GPT-4.1-mini gets 36.0. With Alexein, the same model gets 94.0, ahead of MindMemOS (+ Dreaming) at 92.0 with gpt-5-mini.

  • Alexein
  • Memory and retrieval systems
  • No memory system: a model reading everything, or plain retrieval
  1. AlexeinGPT-4.1-mini

    94.0
  2. MindMemOS (+ Dreaming)gpt-5-mini

    92.0
  3. GPT-4o, whole context

    60.0
  4. HippoRAG 2GPT-4o-mini

    54.0
  5. BM25GPT-4o-mini

    48.0
  6. Claude 3.7 Sonnet, whole context

    43.0
  7. GPT-4.1-mini, whole context

    36.0
  8. Gemini 2.0 Flash, whole context

    30.0
  9. MemGPTGPT-4o-mini

    28.0
  10. CogneeGPT-4o-mini

    28.0
  11. MemoRAGGPT-4o-mini

    21.0
  12. Mem0GPT-4o-mini

    18.0
  13. MIRIXGPT-4o-mini

    14.0
  14. GraphRAGGPT-4o-mini

    14.0
  15. ZepGPT-4o-mini

    7.0

Accuracy on single-hop fact consolidation; each row answers with the model beside it. Alexein’s is self-reported, from 100 questions per fact-consolidation set; published rows from MindMemOS, arXiv 2608.12428 (opens in a new tab) and MemoryAgentBench, arXiv 2507.05257 (opens in a new tab).

On this page

06BEAM

A million tokens a conversation, level with Mem0.

35 conversations of about a million tokens each, every one larger than GPT-5’s 400,000-token window. Alexein hands the answer model a brief instead, and answers level with Mem0.

Level with Mem0

The same 700 questions, GPT-5 answering and judging with Mem0’s published harness (opens in a new tab), no hints.

AlexeinMem0
Mean score63.4964.09
Pass rate70.57%70.14%

Mem0 is ahead on the mean by 0.60, Alexein on the pass rate by 0.43 points: less than a point apart, either way.

Alexein’s figures are self-reported; Mem0’s are from its published harness (opens in a new tab). The benchmark: BEAM, arXiv 2510.27246 (opens in a new tab).

Hindsight (opens in a new tab) and Exabase (opens in a new tab) report BEAM accuracy from their own setups; the only like-for-like accuracy comparison here is the one with Mem0.

On this page

07Not first, yet

Where Alexein isn’t first, yet.

Two benchmarks where other systems’ published scores are ahead of Alexein’s, shown as plainly as the rest. On BEAM, the one like-for-like comparison is level with Mem0; on RHELM, frontier models reading the whole history score higher.

500 questions · accuracy

LongMemEval-S

Ahead of Mem0’s open-source release, run in the same setup. Mem0’s hosted platform reports 93.4 on its blog (opens in a new tab), 1.8 points above Alexein’s 91.6, and other vendors self-report higher still.

  1. Mem0 Platformhosted, Mem0’s blog

    93.4
  2. Alexein

    91.6
  3. Mem0 OSSsame setup

    91.0

1,000 questions · F1

MuSiQue

F1 44.0 (EM 33.9), 4.6 behind HippoRAG 2. Published rows answer with Llama-3.3-70B (HippoRAG 2’s Table 2 (opens in a new tab)).

  • Alexein
  • Retrieval systems
  • Baselines: retrievers alone, and no retrieval
  1. HippoRAG 2

    48.6
  2. NV-Embed-v2

    45.7
  3. GritLM-7B

    44.8
  4. Alexein

    44.0
  5. GraphRAG

    38.5
  6. HippoRAG

    35.1
  7. RAPTOR

    28.9
  8. No retrieval

    26.1

Alexein’s results are self-reported; published results as their sources report them.

On this page

08All results

Every benchmark, in one place.

Alexein’s figure on each benchmark beside the best comparable published ones, then RHELM’s two tables side by side.

Memory benchmarks

  • LoCoMo1,540 questions

    Accuracy, LLM judge

    Alexein
    94.6

    Every row answers with GPT-4.1-mini

    MindMemOS
    94.03
    Mnemis
    93.9
    EverOS
    93.05
  • LoCoMo-Plus401 questions

    Accuracy, binary judge

    Alexein
    55.4

    The paper’s Table 1; its memory systems answer with GPT-4o

    A-Mem
    17.20
    Mem0
    15.80
    SeCom
    14.90
  • LongMemEval-S500 questions

    Accuracy

    Alexein
    91.6

    Mem0 Platform’s figure is from Mem0’s blog

    Mem0 Platform
    93.4
    Mem0 OSSsame setup
    91.0
  • MuSiQue1,000 questions

    F1

    Alexein
    44.0EM 33.9

    Published rows answer with Llama-3.3-70B

    HippoRAG 2
    48.6
  • MemoryAgentBench, fact consolidation100 questions per set

    Accuracy

    Alexein
    94.0

    Alexein answers with GPT-4.1-mini

    MindMemOS (+ Dreaming)gpt-5-mini
    92.0
    GPT-4.1-mini, whole context
    36.0
  • BEAM, 1M tokens700 questions

    Mean score, and pass rate

    Alexein
    63.4970.57% pass

    GPT-5 answering and judging, Mem0’s published harness

    Mem070.14% pass
    64.09

RHELM leaderboard, 1,305 questions, 18 rows

  • Without external sourcesChats only

    Alexein’s average
    35.6
    Rank, of 18
    4th
    Best other memory framework
    memU 23.1
    Alexein’s lead over it, same reader
    +12.5
    Best overall
    GPT-5.5 57.01M-token context
  • With external sourcesChats, emails and documents

    Alexein’s average
    48.5
    Rank, of 18
    3rd
    Best other memory framework
    memU 33.6
    Alexein’s lead over it, same reader
    +14.9
    Best overall
    GPT-5.5 73.31M-token context

On this page

09Methodology & sources

How these were run.

Self-reported
Alexein’s results are the team’s own runs, on every question of each benchmark (on MemoryAgentBench, 100 questions per fact-consolidation set; on BEAM, its 1M-token split). Everyone else’s are as their sources publish them.
Same answer model
The same benchmark answered by the same model, so the memory is what differs: every LoCoMo row answers with GPT-4.1-mini, as do both sides of every same-model pair. Judges follow each source; Alexein’s LoCoMo answers are judged by GPT-4o-mini, majority of three.
LoCoMo’s answer key
An audit (opens in a new tab) lists errors in LoCoMo’s answer key. They affect every top system’s score.
LoCoMo-Plus
Compared with the paper’s own Table 1, where its memory systems answer with GPT-4o. A later paper reports a higher score from its own runs.
RHELM’s margin
±2.6 at 95% on an average.
BEAM
The 1M-token split, run with Mem0’s published harness, GPT-5 answering and judging, no hints. Every conversation is larger than GPT-5’s 400,000-token window.

On this page

Give your agents this memory.

The memory behind these results, a few lines of code away.