MemTensor releases MemT-1.0 models that keep conversation memory inside the network

What MemTensor released

MemTensor has published two large language models that store what they learn during a conversation inside the network itself, instead of re-reading the whole chat history every time they answer. The family is called MemT-1.0. MemT-1.0-Nano has 36 billion parameters, of which 3 billion are active per token, and is built on Alibaba's Qwen3.6-35B-A3B. MemT-1.0-Flash has 288 billion parameters with 17 billion active and is built on DeepSeek-V4-Flash.

The weights went up on Hugging Face on 15 and 17 October 2026, and the technical report, titled "MemT-1.0: Towards Memory-Native Evolution", was submitted to arXiv on 19 October. By 21 October it had more than 140 upvotes on Hugging Face's daily papers page, and the companion GitHub repository had just over 200 stars. Nano's weights are under the Apache License 2.0 and Flash's under the MIT License.

The pitch is a different answer to a problem every chatbot and agent has: how to remember a long history cheaply. Most systems either keep the full text in the context window, which gets slower and more expensive as it grows, or store notes in an external database and retrieve pieces of it. MemT-1.0 instead writes the history into a fixed-size block of numbers that the model reads with its own attention layers.

How native memory works

In selected transformer layers, MemT-1.0 adds a native memory block that runs alongside the model's normal attention. The report describes two parts. A local memory block holds a dynamic memory matrix that starts at zero for each new conversation and changes as the conversation goes on. A hyper memory block holds trained projections and gates that decide what gets written, erased, kept and read. The hyper memory is learned in training and stays fixed when the model is in use.

At the end of each turn, the model writes that turn's hidden states into the memory matrix token by token, using an update rule based on Gated DeltaNet-2 with what the authors call a stable, non-expansive erase step. During generation, a memory attention path reads the matrix, normalises and gates the result, and adds it to the output of ordinary attention. The text of earlier turns does not need to stay in the prompt.

Nano adds memory to 10 of its 40 layers and keeps about 10.5 million numbers of state per conversation, roughly 40 MiB in 32-bit floating point. Flash uses 20 layers and about 336 million numbers, around 1.25 GiB. A technique the team calls Mixture-of-Storage lets it scale that capacity further. MemTensor says MemT-1.0 builds on Metis, its earlier memory model, but replaces Metis's token selection with the token-level update.

Training used 34 billion tokens: 20 billion of continued pre-training and 14 billion of post-training on memory tasks such as summarisation, continuation and compression. The report says part of the corpus is based on anonymised interaction patterns from MemOS, a memory system, and that the backbone weights were frozen while the memory parts were trained.

How MemT-1.0 scores on memory tests

MemTensor tested both models on 11 memory benchmarks, including LoCoMo, LongMemEval-S, MemoryAgentBench and the million-token BEAM test. Among methods that keep memory in model parameters or state rather than in text, Flash averaged 50.6 and Nano 47.0. The best of those parametric baselines, Temp-LoRA, scored 38.1, so MemT-1.0 leads that category by 12.5 and 8.9 points.

The report is open about where it falls short. Text-based memory still wins overall. Simply giving the backbone the full history scored about 65 to 66 on the same average, the MemOS and Mem0 memory systems about 65 to 68, and plain retrieval (DenseRAG) about 57. The model card says strong textual-memory systems "remain ahead overall, particularly on long histories that require detailed evidence retrieval." The abstract's wording is that MemT-1.0 remains "competitive" with retrieval methods.

On general skills, the team filled memory with irrelevant material and compared the models with their own backbones. Flash came out level on average, gaining on BigCodeBench-Hard and losing a little on MMLU-Pro, GPQA Diamond and LiveCodeBench v6. Nano lost about one point on average, mostly on LiveCodeBench v6. All of these numbers are MemTensor's own; AI Lookout did not run the models.

Speed and storage at long histories

The clearer advantage is cost at long histories. On two GPUs with Hugging Face inference, Nano was slower than the full-context baseline for short chats, about even at 32,000 tokens, and about 1.2 times faster end to end at 64,000 tokens, 1.86 times at 128,000 and 3.99 times at 256,000. Time to first token for a new question stayed at roughly 104 to 112 milliseconds across all lengths, while the full-context model rose to about 257 ms at 128,000 tokens and 445 ms at 256,000.

Because the memory state does not grow with the conversation, MemTensor says Nano needs about 66 times less persistent storage than keeping a full key-value cache for a 128,000-token history. That is the same cost problem that tools such as Tensormesh's KV-cache platform try to manage from the infrastructure side; MemT-1.0 tries to remove it inside the model.

Memory for coding agents

The report also tests memory as a way to help coding agents. The team turned 41,057 successful trajectories from the Scale-SWE dataset into a bank of 1,040 reusable "experiences" describing when a technique applies and how to verify it. Before or during a task, a relevant experience is retrieved and either pasted into the agent's prompt or written into Nano's native memory. The idea is close to the learned-memory approach in IBM's ALTK-Evolve, but the experience lives in the model's state rather than in text.

On SWE-bench Verified, with experience stored before execution, the share of tasks solved in all three runs rose from 54.0% for plain Qwen3.6-35B-A3B to 61.6% for Nano with native memory, the same as putting the experience text in the prompt. When experience was added only after the agent hit trouble, Nano reached 62.8% against 61.8% for the text version. On SWE-bench Multilingual, though, the text version did better, 56.3% against 53.7%. The gain comes mostly from the experience itself; native memory roughly matches, rather than beats, the textual approach while keeping the prompt shorter.

How to try it, and the licences

Both models load through Hugging Face Transformers with custom code enabled. The model card shows a session interface: turning on memory for a turn saves it into the latent state and clears the plain text, and the state can be saved to a file and restored later without the original conversation. In MemTensor's example, a restored session with zero stored messages still answers a question about a kitten described earlier. Standard chat without memory also works.

Hardware is the practical hurdle. Nano is a 36-billion-parameter mixture-of-experts model, and the card asks for Python 3.11, PyTorch 2.11 built for CUDA 13 and recent Transformers. Flash, at 288 billion parameters on top of DeepSeek's V4 Flash line, is a multi-GPU server model.

The licences need a careful read. The weights are Apache 2.0 (Nano) and MIT (Flash), which allow commercial use. The GitHub repository, which holds the SGLang serving integration, is under the PolyForm Noncommercial License, and its README says commercial use of that software is not permitted. The report itself is under CC BY-NC-SA 4.0. Companies that want the faster serving path would need to talk to MemTensor.

Limitations

The authors list the main weakness themselves: a fixed-size memory has fixed capacity. As a conversation grows, new information has to be squeezed into the same state, which trades away earlier details. That is why text-based memory still wins on tests that need exact evidence from long histories. The memory is also not shared between conversations; each trajectory starts from an empty state.

All benchmarks, speed tests and agent results are self-reported, run on MemTensor's own harnesses, and have not been reproduced by outside groups yet. Memory that cannot be read as text is also harder to inspect or edit than a notes file, a contrast with systems such as Tencent's HY-Memory, which writes memories as text that people can review.

Why it matters

Long-running agents and assistants are pushing context windows and memory databases hard, and both get more expensive as histories grow. MemT-1.0 is one of the first open-weight attempts to make memory part of the model's forward pass at a large scale, with a constant-size state and flat response times as history grows.

Its results show both sides of that trade. It is clearly better than earlier parametric-memory methods and faster at very long histories, but it does not yet match good text-based memory on accuracy. For builders, the practical question is whether a 1.9 to 4 times speedup at 128,000 to 256,000 tokens is worth some recall loss, and whether the noncommercial serving code fits their plans.