Memory recall. Measured, not modelled.
Hit-rate on a standardized 100-question eval set, plus an honest comparison against vendor-reported numbers. Recall latency across larger corpora is not measured yet, so no Zephr latency figure is shown. The methodology and limitations are disclosed before the results, because a benchmark without them is a marketing number.
Methodology and limitations
Every number below is sourced from the committed results artifact (packages/evals/src/memory/results/2026-09-03.json). No figures are hand-typed. The semantic recall path is not measured; only the lexical/BM25 path is benchmarked. Recall itself is zero-price at hosted pricing (retrieval produces no chargeable event); capture, indexing and storage are the billable operations.
Eval set
Standardized 100-question set with one planted gold answer per question, across 10 sessions. Questions are committed, not generated at runtime.
Scoring
Hit-rate@5: does the gold answer appear in the top 5 results? Precision@k and MRR come from the same ranked results.
Limitations
No distractors in the corpus. Substring grading (exact match of gold in returned text). Semantic vector path is not measured (lexical/BM25 only). The corpus is synthetic (10 sessions × 10 beliefs), not a production traffic pattern, and recall latency across larger corpora is not measured — this page publishes hit-rate only.
Source: packages/evals/src/memory/results/2026-09-03.json · generated 2026-09-03 · schema: zephr.memory-bench.v1
node scripts/memory-bench/run.mjs --out-dir reports/memory-bench22.0% hit-rate@5. Room to grow.
100 questions across 10 sessions, recall limit of 5. These are baseline numbers on a synthetic corpus — not production traffic. We publish them because measured baselines are more useful than marketing claims.
22.0%
hit-rate@5
Fraction of 100 questions where the gold answer appears in the top 5 results.
8.0%
precision@1
Fraction of questions where the gold answer is the very first result.
13.8%
MRR
Mean reciprocal rank of the gold answer across all questions.
7.3%
precision@5
Precision at 5: how many of the top 5 results are the gold answer.
Values sourced from the committed results artifact, not hand-typed. The artifact is reproducible from a fresh checkout via the command above.
Vendor-reported numbers. Not reproduced by us.
The table below shows competitor-reported benchmark numbers from their own publications. These were not measured on identical hardware or corpora. Each row cites its primary source. We do not reproduce their methodology — we publish our own numbers above and let readers compare.
| Vendor | Metric | Value | Status | Source |
|---|---|---|---|---|
| mem0 | LongMemEval | 94.4 | vendor-reported, not reproduced by us | mem0.ai/blog/state-of-ai-agent-memory-20… |
| mem0 | LoCoMo | 92.5 | vendor-reported, not reproduced by us | mem0.ai/blog/state-of-ai-agent-memory-20… |
| Zep | LongMemEval | 63.8 | vendor-reported, not reproduced by us | arxiv.org/html/2501.13956v1 |
| Letta | LongMemEval | not evaluated | vendor-reported, not reproduced by us | no primary source |
mem0 (LongMemEval): Self-reported by mem0 (vendor claim). Independent evals report lower (vectorize.io: 49.0%). Mem0 v3 algorithm, 500 questions, top-200 retrieval.
mem0 (LoCoMo): Self-reported by mem0 (vendor claim). 1540 questions, ~6956 tokens/query average.
Zep (LongMemEval): From Zep paper (Graphiti), GPT-4o model. Self-reported. Note: mem0 v3 reports 94.4% on same benchmark — different versions/algorithms may explain gap. Zep paper also reports 71.2% with gpt-4o on a different configuration.
Letta (LongMemEval): No primary source found. Letta (formerly MemGPT) is a stateful agent runtime, not a recall API. Their own benchmark (Letta Leaderboard, May 2025) evaluates LLM memory management capability, not recall hit-rate. No comparable published LongMemEval score exists.
All competitor numbers are vendor-reported and were not reproduced by Zephr on identical hardware or corpora. Source URLs and retrieval dates are in the data file (packages/evals/src/memory/comparators.json).
Close
Evidence you can inspect.
Every number on this page traces to a committed artifact. The methodology and limitations are disclosed before the results, and the comparison table cites primary sources. The benchmark is reproducible from a fresh checkout.