Agent Memory · Cost-Aware Systems

Memory without
calling the model.

Most agent memory systems pay LLM tokens to write summaries and score retrievals. Zero-Mem asks whether any of that generation is necessary, and finds the answer is no. Raw traces, two structural indexes, one deterministic combination step. The model is called once: for the final answer.

Core result
57.6% lower memory-operation time cost versus the fastest baseline, at matched final-answer quality, with zero LLM tokens consumed on memory operations.

First surfaced in Tandemly Briefing — 2026-08-05.

scroll to explore
01The problem

Memory has a
hidden token bill.

Agent memory sounds straightforward: record what happened, retrieve what matters. In practice, most systems reach for the language model to do both jobs, and the cost accumulates quietly.

A typical agent memory system works something like this: when a conversation ends, call the model to write a summary. When a question comes in later, call the model again to score which stored summaries might be useful. Both choices feel reasonable. Summarization compresses long histories into something searchable. Retrieval scoring seems to need natural language understanding to do well.

Neither of those assumptions is quite right, and in a long-running agent system the cost of getting them wrong accumulates fast. Every write-time summarization call and every read-time retrieval-scoring call consumes LLM tokens. In systems handling hundreds or thousands of conversational turns, those maintenance calls can rival the cost of the actual task the agent is doing. The memory system becomes a significant fraction of the token bill, quietly.

The deeper problem is structural. A summarization call doesn't just cost tokens; it also throws information away and introduces the model's own biases into the stored representation. A retrieval-scoring call is doing a matching job: which stored memory is most relevant to this query. That looks like a problem that doesn't need generation to solve. It looks like a data structure problem.

The question Zero-Mem asks

Does structured agent memory actually need LLM generation? No step outside final question-answering should invoke a model or consume LLM tokens. Is that achievable without sacrificing answer quality?

02The solution

Store the trace.
Skip the generation.

Zero-Mem replaces every generation step in the memory pipeline with deterministic structure. Two structural views over raw traces, combined per-query without an LLM call.

The starting move is simple: don't generate a representation, store the raw trace. Raw interaction records are preserved as-is when a conversation ends. This immediately eliminates the write-time LLM call and the information loss that goes with it. The engineering work moves to the retrieval side: how do you make raw traces efficiently searchable without calling the model?

Zero-Mem builds two structural views over the stored traces, each capturing a different dimension of relevance.

1
Entity-context graph
Every entity mentioned across conversations (people, topics, objects, locations) becomes a node. Edges connect entities that appeared together in the same interaction, weighted by co-occurrence context. When a query arrives, the graph lets the system navigate entity relationships to find relevant stored interactions, without calling the model to score text similarity.
2
Temporal hierarchy
Interactions are organized by recency: recent exchanges sit at the top of the hierarchy, older ones lower. This captures conversational locality. What happened moments ago is more likely to matter than what happened weeks ago, and the hierarchy encodes this without the system needing a model to figure it out per query.
3
Deterministic calibration
When a query arrives, both views score it independently. A calibration step weighs the two scores together, resolving conflicts by downweighting contradicted evidence. No model call. The winning traces pass to the QA reader. That reader is the only LLM call in the entire memory loop.
Standard memory system
LLM at every step. Write: call model to summarize. Index: call model to embed or categorize. Retrieve: call model to score candidates. Answer: call model for QA. Four or more model calls per memory cycle.
Zero-Mem
LLM at one step only. Write: store raw trace (no model call). Index: build entity graph and temporal hierarchy (no model call). Retrieve: deterministic calibration of both views (no model call). Answer: QA reader. One model call per memory cycle.
What "deterministic" means here

The calibration step doesn't learn or generate; it combines two structural scores by a fixed rule and resolves disagreements through downweighting. This makes the memory system predictable: given the same query and the same stored traces, it produces the same retrieval output every time. No variance from sampling, no generation artifacts.

03Findings

Faster memory,
same answers.

The matched-quality result is what makes the efficiency gain credible. Zero-Mem doesn't trade answer quality for speed. It removes generation from the maintenance path and the answers stay the same.

Memory-op time cost reduction
57.6%
vs. fastest baseline, at matched quality
LLM tokens on memory ops
0
write, index, and retrieve steps
LLM calls in memory loop
1
final QA reader only
The quality match is the finding

If Zero-Mem were faster but less accurate, it would be a straightforward quality-cost tradeoff and the decision about whether to adopt it would be context-dependent. The matched-quality result removes that tradeoff: the memory infrastructure becomes invisible in terms of token cost, without changing what the agent can answer.

This is a different kind of efficiency claim than most cost-reduction work makes. It isn't "accept worse answers to save tokens." It is "these generation steps were not adding value in the first place."

Where Zero-Mem sits in the memory landscape

Recent memory work has moved in several directions at once. Delta-mem modifies the attention architecture to extend the effective context window inside the model. NapMem reframes retrieval as agentic navigation across a multi-granularity memory structure. MemSyco-Bench identified a failure mode where memory systems increase sycophancy relative to no-memory baselines.

Zero-Mem operates on a different axis from all of these: it targets the generation calls in the maintenance path, not the retrieval strategy or the model architecture. These approaches are complementary. An agent could use Zero-Mem's trace storage pattern under a NapMem-style navigation layer, for example, or audit memory sycophancy in a Zero-Mem system. The approaches don't compete; they cover different parts of the memory stack.

Scope and honest limits

The 57.6% figure is real and the matched-quality claim is meaningful, but the specific benchmarks and task distributions used for evaluation are not described in the abstract-level summary available for this synthesis. How much the result generalizes across memory-heavy versus mixed workloads, and across different entity density and conversation length profiles, matters for applying it in production. Test on your own workload before treating the result as universal.

04Practical takeaways

What to do
with this.

The core insight generalizes beyond Zero-Mem specifically: any time you use a language model where a deterministic operation on a proper data structure would produce the same output, you're paying generation cost for a retrieval or matching job.

1
For developers building agent memory
Map every model call in your memory pipeline and categorize it: is this doing generation (producing new content) or retrieval (finding relevant existing content)? Write-time summarization and read-time retrieval scoring are usually retrieval jobs dressed up as generation. Those are the candidates to replace with structural approaches.
2
For anyone paying attention to token costs
Memory maintenance is a quiet cost center in long-running agents. It doesn't show up on a per-request basis as obviously as the QA step does, but in high-volume or long-context deployments it accumulates. Zero-Mem's result suggests the maintenance cost can be reduced to near zero without affecting answers. That's worth auditing even if you don't adopt the specific approach.
3
For memory system designers
The entity-context graph and temporal hierarchy are a specific design choice. The more general principle is: don't ask the model to produce a searchable representation when a structural one will do. Graphs for relational retrieval, time hierarchies for recency retrieval, and deterministic combination for query-time resolution are each independently useful patterns. You don't have to adopt all three at once.
4
A note on the research frontier
Zero-Mem makes a strong bet: the right primitives for agent memory are graphs and time structures, not latent embeddings generated by a model. That's a substantive architectural claim. The result is promising and the reasoning is sound, but the paper is recent, the code is pending public release pending peer review, and the benchmark scope is limited to what's described in the abstract. Treat it as a strong hypothesis to test, not a settled answer.
05Getting started

Where to go
from here.

The path from reading this to testing it in a real system is shorter than most research papers suggest.

1
Audit your memory pipeline for LLM calls
Map every point where a model is called outside the task-answer step. Write-time (summarization, indexing) and read-time (relevance scoring, candidate ranking) calls are the ones Zero-Mem eliminates. Categorize each and estimate how many tokens each category consumes per 1,000 conversational turns in your system.
2
Run a simple replacement experiment
Pick one write-time LLM call and replace it with a raw trace log plus a simple entity-extraction lookup (a named entity recognizer, not a model call). Measure retrieval precision on a sample of test queries before and after. If quality holds, you have evidence that the generation step wasn't earning its cost on your workload.
3
Read the paper
Xiao, Zhu, Zhang et al. "Zero-Mem: Zero-Token Memory Operations for LLM Agents." arXiv:2607.29377. The abstract and methodology sections give the structural detail needed to implement the entity-context graph and temporal hierarchy components.
4
Check the GitHub repository
The official implementation is at GitHub (TheMoon0815/Zero-Mem). At the time of synthesis, code release is pending peer review. A community reimplementation is available at woolcoxm/zero-mem-pi (TypeScript, pi coding agent) and ptaranat/zeromem (Rust, Hermes Agent memory provider) if you want to explore the approach sooner.
5
Measure quality at each step
Zero-Mem's matched-quality claim is what makes the efficiency gain credible. Any replacement of a generation step with a structural one needs the same validation on your workload: run both approaches on the same test set and confirm final-answer quality holds before committing to the structural approach in production.