Agent memory evaluation guide

Evaluate memory, not demos.

Start with the failure that could hurt the task, choose a suite that exposes it, then compare quality and system cost under the same run contract. A benchmark score without its baseline and operating envelope is not an architecture decision.

Choose the failure to expose
Compare honest baselines
Report the whole system

Failure before suite

Choose the failure to expose

Pick the narrowest suite that can falsify your memory claim. Compose suites only when the product actually spans multiple memory jobs.

Eight evaluation questions mapped to primary benchmark papers. On small screens, scroll the table horizontally.
Benchmark	Ask it when	What it isolates	Boundary	Evidence
LongMemEval	Can a chat assistant retain, revise, and use information across sessions?	Information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.	Pair with system-cost and no-harm measures; the suite does not supply them.	Primary paper Catalog results
LoCoMo	Can the system reason over very long, temporal, and causal dialogue?	Question answering, event summarization, and multimodal dialogue generation on human-verified conversations.	Does not by itself test online forgetting or operation-level write errors.	Primary paper Catalog results
MemoryAgentBench	Can an interactive agent manage memory as information arrives incrementally?	Accurate retrieval, test-time learning, long-range understanding, and selective forgetting.	Use a separate failure-localization suite when you need to find the bad stage.	Primary paper Catalog results
PersonaMem	Can the assistant track changing traits and preferences?	Dynamic profile inference, preference evolution, and personalized response selection across interaction histories.	Measures personalization, not broad factual or procedural memory quality.	Primary paper Catalog results
MemoryBench	Can the system learn continuously from accumulated user feedback?	Service-time continual learning across multiple domains, languages, and task types.	Report retrieval and update cost separately from task quality.	Primary paper Catalog results
HaluMem	Where do fabricated, conflicting, missing, or wrong memories enter the system?	Operation-level errors in memory extraction, updating, and question answering.	Localizes hallucination behavior; it is not a complete personalization test.	Primary paper Catalog results
BEAM	Does memory behavior survive coherent histories beyond ordinary context sizes?	Long, topically diverse conversations with questions spanning multiple memory abilities and contexts up to ten million tokens.	Hold context budget and compute fixed; scale can otherwise hide the trade-off.	Primary paper Catalog results
HELMET	Is the bottleneck the memory system or the long-context backbone?	A long-context control covering seven application-centric categories at controllable lengths.	This is a model-context control, not an agent-memory benchmark.	Primary paper Catalog results

Ablate before claiming

Compare honest baselines

A memory layer earns its complexity only when it beats simpler ways to expose the same information. Run all feasible baselines under one locked comparison contract.

00 · Control

No memory

Use the same model and task prompt with no retained history or retrieved state.

01 · Context baseline

Full context

Provide all allowed history when it fits. Record truncation when it does not.

02 · Simple comparator

Simple retrieval

Use fixed chunking, embedding model, search policy, and top-k without consolidation.

03 · Treatment

Candidate system

Run the proposed write, evolution, retrieval, and reading policies unchanged.

Quality, cost, and harm

Report the whole system

Publish the benchmark's task metric, then the operating cost and failure surface that a user will experience. Report distributions and failure counts, not only averages.

Outcome and safety

Did memory improve the task?

Task-grounded outcome: suite metric, scorer, and version.
Failures and retries: attempted, completed, failed, retried.
No-harm cases: irrelevant, conflicting, poisoned, private, and deleted memories.
Evidence: answer-to-memory provenance when the task supports it.

Performance and footprint

What did memory cost?

Write or indexing latency (p50 / p95).
Recall latency (p50 / p95).
End-to-end latency (p50 / p95).
Storage footprint at a fixed history size.
Cold-start time before the first valid result.
Token or compute cost: total tokens = input tokens + output tokens.

Localize the failure before changing the architecture

Formation

Missing, fabricated, overbroad, or privacy-unsafe writes.

Evolution

Stale facts, unresolved conflicts, lost corrections, or failed deletion.

Retrieval

Misses, irrelevant hits, poisoned evidence, or provenance mismatch.

Reading / action

Correct memory is ignored, misread, or applied to the wrong decision.

System

Timeouts, rate limits, retries, storage errors, or cost-budget breaches.

Minimum evaluation receipt

Make every result inspectable

run:
  harness_commit: <sha>
  dataset_and_scorer: <name / version>
  model_and_prompt: <provider / model / version / prompt hash>
  hardware_or_region: <description>
  attempts_completed_failed: <counts>
result:
  task_metric: <metric / value / uncertainty>
  write_recall_e2e_latency: <p50 / p95>
  storage_and_cold_start: <bytes / time>
  input_output_total_tokens: <counts>
  no_harm_failures: <cases / count>

Ship gate

Ship memory only when it improves the task over the simplest viable baseline without breaking the latency, cost, privacy, deletion, or no-harm budget.

Keep the guide falsifiable

Found a missing evaluation question?

Propose the memory behavior, a primary benchmark source, the decision it changes, and the limitation that prevents misuse. Aggregate scores without a reproducible harness do not belong here.

Suggest a benchmark Read the evidence rules

Primary benchmark papers

These papers define the benchmark tasks and claims summarized above. They do not endorse this guide or make results from different run contracts comparable.