Eragon MemBench: measuring memory for work

Today we’re introducing Eragon MemBench, a benchmark for how well AI assistants remember things at work. It covers the hard cases: several people sharing context, conversations full of tool output, decisions that change, and information that should never be kept.

We compared Eragon with two widely used open-source memory systems: Graphiti (the engine behind Zep) and Mem0. On MemBench’s harder scenarios (hard-v1), Eragon’s memory system achieved the highest accuracy, lowest latency and lowest token use. Eragon answered 75.3% of questions correctly, nearly double Mem0’s 37.8%. It returned answers in a median of 5.4 seconds, 1.8 seconds faster than Graphiti, and passed the answering model about 22,000 tokens per answer, 22% fewer than Graphiti and 46% fewer than Mem0.

Below, we describe what MemBench tests, how the three systems performed, and how Eragon’s memory works.

Why agent memory in real enterprise environments is hard

In production enterprise environments, agent memory systems can’t just retrieve the sentence that most resembles a question. An account owner changes. A rule gets superseded. Two colleagues share a first name. A document contains text that looks like an instruction to the assistant but should never govern its behavior.

Workplace memory also requires updating information, not just accumulating it. We sampled two weeks of Eragon’s production memory saves in September 2026:

Most benchmarks focus on how memory systems handle new facts. Eragon MemBench also tests how they handle facts that change within complex enterprise environments.

One scenario, several ways to fail

Consider an analytics team working on a revenue dashboard. Over several weeks:

• The orders table is replaced.
• A dashboard owner is named.
• A nightly job moves to a new start time.

Along the way, three traps appear. Another dashboard is owned by someone with the same first name. A password gets pasted into chat. A glossary page tells AI assistants to report an outdated metric definition.

Weeks later, in a new session, someone asks: “The weekly net revenue tile on Revenue Pulse needs a new filter. Who do I go to?”

The right answer is Tamsin Okoye, the owner of Revenue Pulse, not the other Tamsin. A different question might ask for the job’s current start time rather than the old one. Neither the password nor the glossary’s instruction should become memory just because it appeared in the history.

Every scenario is evaluated the same way:

1. Replay dated conversations and tool output in order.
2. Let the memory system decide what to store, update or discard.
3. Add background memories, including lookalike people and outdated facts.
4. Weeks later, ask questions in fresh sessions, without repeating the earlier context.
5. Grade the answers, and separately audit what the system stored.

What MemBench covers

The full suite has 55 scenarios across 10 business functions, with 738 questions of 10 types. Each type targets a different way memory fails at work:

About half of all conversation turns (51%) contain tool output, as real work does.

MemBench also tests what a system chooses to store. The suite has 1,289 annotated items marking what is worth keeping and what isn’t. Of these, 571 should never be stored, including credentials, other people’s private details, expired task state, speculation and instructions embedded in documents.

MemBench hard-v1, the harder subset of scenarios and questions, contains 80 planted lookalikes and stale facts. All people, companies, systems and figures are fictional; no customer data is used.

Results

When testing Eragon’s memory system, we wanted to push our architecture to its limits, so we focused the evaluation on MemBench hard-v1, its 10 hardest scenarios. Each memory system (Eragon, Graphiti and Mem0) started with 100 background memories per person and answered the same 86 questions. We ran each evaluation four times and report the mean.

Eragon achieved the highest accuracy of the three, answering 75.3% of questions correctly, ahead of Graphiti (68.3%) and nearly double Mem0 (37.8%).

Accuracy isn’t the whole experience. A memory system that is correct but slow, or that retrieves context inefficiently using many tokens, is harder to use in production. So we also measured two other things:

• Answer time: retrieval plus the model’s response
• Reader input: the number of tokens sent to the answering model

Eragon had the lowest latency and the smallest reader input of the three. Against Graphiti, it was 1.8 seconds faster at the median and used about 22% fewer input tokens. Against Mem0, it was 2.2 seconds faster and used 46% fewer tokens.

Accuracy by type of memory

Beyond overall accuracy, we also tested retrieval on specific kinds of memory tasks, such as applying a remembered rule or following a fact after it changes.

The gaps are widest on questions where the closest-matching sentence isn’t the answer:

• Multi-hop: Eragon 68.8%, Graphiti 31.3%
• Updated facts: Eragon 78.6%, Mem0 19.6%

How Eragon’s memory works

A common approach to agent memory is to store everything and search it at question time. Eragon does most of its work earlier, when a memory is written, so there’s less to sort out when someone asks a question.

Memory is curated, not accumulated. After each conversation, Eragon decides what’s worth keeping and leaves out what should never be stored. When a fact changes, Eragon revises the memory it already has instead of saving a second version next to the first. Every memory records when it became true and who it belongs to: a person, a team or the organization.

Retrieval is cheap because curation already happened. For a new request, Eragon has no pile of conflicting versions to sort through and no transcripts to summarize. It finds the few current facts that apply, filters them to what the person asking is allowed to see, and passes them to the model.

That design drives all three results:

• Accuracy: Each fact has one current, dated version, so the model answers from what’s true now, not from older text that happens to match the question.
• Latency: The expensive work runs after the reply, not while someone waits.
• Token efficiency: The model gets a handful of distilled facts, not the history they came from.

Privacy, scope and forgetting

Enterprise memory has to respect who information belongs to and what shouldn’t be kept. MemBench checks three things:

• whether private details stay out of a colleague’s answer
• whether the assistant uses the right person’s or team’s context
• whether a request not to keep something is honored later

In Eragon’s runs:

We will keep improving how Eragon respects enterprise privacy requirements. We have found this ability essential to supporting complex deployments around the world.

How we ran the comparison, and its limits

Every system passed its memories to Claude Sonnet 5.5 with the same answer prompt, and GPT-6.1-sol graded every answer.

Each memory system used its own published configuration:

• Mem0 ingested conversations as message pairs and searched with top-k 200.
• Graphiti used its quickstart search.

This comparison has two limits:

1. It used 86 of MemBench’s 738 questions and a single background load (100 memories per person). The full suite ranges from 0 to 1,000.
2. We report means and intervals across four runs, but not paired significance tests for every difference.

What’s next

We will keep pushing our architecture to improve memory retrieval across every kind of enterprise task, including those shaped by data privacy requirements. Next, we will measure how systems write to memory: whether they keep useful facts, update stale ones, and leave out secrets, injected instructions and short-lived task state.

We are particularly excited about memory sharing between individual Eragon agents. It lets people collaborate more efficiently, and we will rigorously build and evaluate these capabilities, both in Eragon and across other memory systems.

If you’re interested in learning more about MemBench and Eragon’s memory system, reach out at hello@eragon.ai.