Letta's FrontierMemory benchmark scores agent memory by the work it improves

What Letta released

Letta, the company behind the open-source MemGPT project and the Letta Code agent, on 21 October 2026 released FrontierMemory 1.0, a benchmark that grades AI agent memory by whether it helps the agent do later work. Letta's announcement thread on X describes it as "a new way to evaluate agent memory through downstream task performance", and the launch post on the FrontierMemory site calls it the first benchmark that measures the quality of memory rather than recall of facts.

The setup is simple to describe. An agent is given a pile of transcripts from earlier attempts at related work, some successful and some not. It has up to six hours to turn them into memory: notes, rules or reusable skills. A second agent then tackles new tasks with that memory loaded, and the score is how many tasks it solves.

The first leaderboard puts Anthropic's Opus 5.5 and Zhipu's GLM-5.3 joint first at 73.3%, ahead of Kimi K3 at 66.7%, with OpenAI's GPT-6.1 Sol and Google's Gemini 4 Argon at 60.0%. Meta's Muse Wonder scored 46.7% and xAI's Grok 4.7 33.3%. Among complete agent systems, Letta's own Letta Dream led at 73.3%, followed by Prime Intellect's Prime Agent at 60.0%.

Why recall tests stopped being useful

Memory has become a standard feature of AI agents this year. Letta's post notes that nearly all major coding and personal agents now keep some form of file-based memory that they edit themselves, and that several products, including the Claude Agents SDK, Letta Code and Cognition's Devin, support "dreaming": a background pass that studies past sessions and updates memory between them.

The benchmarks have not kept up. Popular tests such as LoCoMo and LongMemEval quiz an agent on facts from earlier conversations, such as a user's birthday or hobbies. Letta argues that these tests are now largely saturated, because modern coding harnesses can simply search their files with shell tools, and that they never measured the thing users care about: whether an agent gets better at its job over time.

The benchmark arrives in a crowded week for agent memory. Tencent has released Hy-Memory-30B-A3B, a small model trained specifically to write agent memory, and IBM's ALTK-Evolve recently topped a customer-service benchmark using learned memory. FrontierMemory offers a way to compare approaches like these on the same tasks.

How FrontierMemory works

Letta's researchers started from difficult tasks in public coding and knowledge-work benchmarks, including DeepSWE, SkillsBench, AutomationBench, APEX-Agents and EnterpriseOps-Gym, and rewrote them so that solving them requires knowledge that only appears in earlier transcripts. The post compares this to the difference between a brand-new intern, who needs everything spelled out, and a senior employee who has learned the context over time.

The tasks test three kinds of memory. Task memory covers conventions the final prompt leaves out, such as which files a Word2Vec pipeline should skip. User memory covers preferences learned from feedback, such as how a changelog should be formatted. Environment memory covers facts about the workspace, such as where a configuration file with grid limits is stored. For each task, Letta wrote a "gold memory" to confirm the task can be solved with the right notes.

The transcripts are deliberately noisy. Agents get 665 trajectories, of which only 219 are relevant to the test tasks. They come from HarborIndex and DeepSWE, from runs of harnesses such as Codex, Claude Code and OpenHands, and from simulated users who give feedback. Web browsing and looking up solutions are not allowed.

Letta also checked its sources. At least two researchers reviewed each task, and the post says 53% of the original tasks that were kept had broken verifiers: graders that checked for things the task never asked for, rejected correct answers over formatting, or required actions the tools did not allow. Those were repaired. The finished benchmark has 15 tasks.

Which models and harnesses did best

For the model ranking, each model writes memory inside the Letta Dream harness, and that memory is then used by a task agent running in Letta Code. The task agent is run with both Opus 5.5 and GPT-6 Sol at five reasoning-effort levels, and the leaderboard reports the better result at maximum effort.

Cost varies widely. GPT-6.1 Sol reached 60.0% at a task cost of $0.77, the lowest on the board, while Opus 5.5 cost $2.80 for its 73.3% and GLM-5.3 $3.58. Muse Wonder was the most expensive at $4.09 per task for 46.7%. For teams that care about cost, GPT-6.1 Sol offers the most memory quality per dollar in this first release.

The harness comparison uses each product's own dreaming and task tools: Letta Dream with Letta Code, Claude Managed Agents dreaming with Claude Code, the OpenAI Agents SDK's dreaming feature with Codex, Prime Agent's "refine" feature and Hermes Agent's "learn" feature. With Opus 5.5, Letta Dream scored 73.3%, Prime Agent 60.0%, Claude Managed Agents 33.3% and Hermes Agent 26.7%. With GPT-6.1 Sol writing memory, Letta Dream scored 60.0%, the OpenAI Agents SDK 40.0%, Hermes 20.0% and Prime Agent 13.3%. Letta built both the benchmark and the top harness, which readers should keep in mind. GLM-5.3's strong showing adds to recent interest in that model, including Apollo Research's study of its reasoning.

What the memory files reveal

The most useful part of the release may be its examples of good and bad memory. Letta found that instructions work better than summaries. In one task, Opus 5.5 wrote a skill for a fictional company called Brightpath Software whose description told the agent exactly when to use it: "Use when a task names Brightpath or these files." The task agent loaded it. GPT-6.1 Sol split the same knowledge into four skills with more generic descriptions, and the task agent never loaded the relevant one.

Overfitting was the other common failure. Learning from the same transcripts, Opus 5.5 wrote a general rule: "Some instances hold the method only in Mail, so check both." Gemini 4 Argon wrote a rule that only applied if a notes file was "the 6,922-byte stub instead of the 10,294-byte full version". Because it was tied to a file size rather than the task, it never triggered on a new task.

Letta also found no sign that memory written by one model works best only for that model. Opus 5.5's memory helped most even when GPT-6 Sol was doing the work, and the ranking was roughly the same with either task model. That suggests memory written as plain files can move between agents, which matters for anyone designing what an agent should remember.

How developers can use it

FrontierMemory is published as a leaderboard and a write-up at frontiermemory.com. Letta invites submissions of new models and harnesses; a harness must have its own memory system that creates memories or skills between sessions. Groups that want to run the benchmark or contribute are asked to contact Letta's team by email.

For developers building agents, the practical lessons are already clear from the examples. Write memory as instructions with explicit triggers, keep rules general rather than tied to one run, and test memory by whether a fresh agent does better with it, not by whether it can repeat facts. Tools with inspectable memory, such as OpenBMB's PilotDeck 2.0, make that kind of checking easier.

Limits of a 15-task benchmark

Fifteen tasks is a small benchmark. One task is worth 6.7 percentage points, so the gap between 73.3% and 66.7% is a single task, and several models are tied. The release does not report repeated runs or error bars, so small differences should not be read as firm rankings.

Letta has an interest in the outcome. It designed the tasks, wrote the gold memories, ran the evaluations, and its own harness tops the harness table; by default every model is tested inside Letta's tools. The tasks are adapted versions of public benchmarks that Letta modified, which makes independent reproduction depend on Letta publishing the task set and harness code. The post does not say when or whether that will happen.

The first release also covers only one style of memory: offline dreaming over a fixed set of transcripts. Letta says future versions will add longer-horizon tasks, memory that changes the harness itself, and online memory that is updated while an agent works. Its X thread says the next release will include tasks that enable recursive self-improvement from memory.