Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
A new study by researchers at Stanford University and Nvidia proposes a way for AI models to continue learning after deployment, without increasing inference costs. For enterprise agents who must digest long documents, tickets, and logs, this is an attempt to achieve “long memory” without paying attention to the costs that increase with context length.
The approach, called “End-to-end training on testing times” (TTT-E2E), reframes language modeling as a continuous learning problem: instead of memorizing facts during pre-training, models learn to adapt in real time as they process new information.
The result is a Transformer capable of matching the accuracy of full-attention models over a long context while operating with near-RNN efficiency – a potential breakthrough for enterprise workloads where context length collides with cost.
The precision-efficiency trade-off
For developers building AI systems for tasks involving long documents, choosing model architecture often involves a painful tradeoff between accuracy and efficiency.
On one side are Transformers with full personal attention, currently the gold standard for precision. They are designed to loop through the keys and values of all previous tokens for each new token generated, providing them with lossless recall. However, this precision comes at a high cost: the calculation cost per token increases considerably with the length of the context.
On the other side are linear temporal sequence models, which keep inference costs constant but struggle to retain information over very long contexts.
Other approaches attempt to split the difference – sliding-window attention, attention-recursion hybrids, and other efficiency tricks – but they still tend not to give full attention to hard language modeling.
The researchers’ bet is that the missing ingredient is compression: instead of trying to recall each token exactly, models should distill what matters into a compact state.
Training during testing period
The main innovation of the paper is the application of training to test time (TTT) to language modeling. This transforms the model from a static database into a flexible learner.
In standard AI deployment, models are trained to minimize losses and then deployed as frozen artifacts. If you try to make a static model learn during deployment, it usually performs poorly because it was never trained to update efficiently.
Researchers solve this problem by moving from standard pre-training (teaching the model facts) to meta-learning (teaching the model how to learn). The objective is to optimize the model "initialization" so that it can quickly absorb new information when it comes online.
The process involves simulating inference time learning during the training phase:
-
Inner loop (learn): During training, the model treats the text as a stream and makes small, temporary updates as it predicts the next token, simulating how it would adapt during inference.
-
Outer loop (teach him how to learn): The system then updates the model initialization so that the next round of streaming adaptation becomes faster and more accurate.
Although the idea of a model changing its weighting during deployment may seem risky to business executives concerned about reliability, co-author Yu Sun says it is mathematically safer than it seems.
“You should think of the model as an RNN with a huge hidden state,” says Sun. He notes that if an enterprise feels secure deploying standard Transformers or RNNs, TTT’s stability profile is comparable.
Dual memory architecture
To implement TTT-E2E, researchers modified the standard Transformer architecture to support this new learning paradigm, creating a hierarchy that separates cheap management of short-term context from selective updates of long-term memory.
-
TThe model uses sliding window attention rather than total attention. This acts as the model "working memory," looking only at a fixed window of recent tokens to handle immediate syntax and local references. This ensures that the cost of processing a new token remains constant rather than increasing as the context expands.
-
The model uses “targeted weight updates.” While standard models have weights completely frozen when used, TTT-E2E designates specific sections (multi-layer Perceptron layers in the final 25% of model blocks) that are editable.
-
The architecture uses “dual-track storage” to prevent the model from forgetting general education while learning a new document. Each updatable block contains two MLP components: a static layer that contains pre-trained general knowledge and a dynamic layer that updates in real time to store the context of the current document.
The innovation lies in how the model handles the information that comes out of the sliding window. In a standard sliding window model, once a token slides out of sight, it is forgotten. TTT-E2E prevents this via compression. As the window moves, the model uses the prediction of the next token to "compress" the passage of information directly into the weights of the dynamic MLP layers. This consolidates the gist and facts from the earlier parts of the document into the template structure, serving as a long-term memory.
TTT-E2E in action
The main result: TTT-E2E continues to improve as context length increases – matching or exceeding full attention – while effective baselines plateau after around 32,000 tokens.
To validate their approach, the researchers trained models ranging from 125 million to 3 billion parameters. They used a two-step training process: pre-training on contexts of 8,000 tokens and fine-tuning on contexts of 128,000 tokens. These models were tested against robust benchmarks including Full Attention Transformers, Sliding Window Attention Transformers (SWA), hybrid models (Mamba 2 and Gated DeltaNet), and TTT-KVB (an earlier form of training at the time of testing).
The results highlight a significant advance in scaling. The most critical experiment tested performance, with the input document increasing from 8,000 to 128,000 tokens. The Full Attention Transformer, the benchmark, continued to improve its performance (less losses) as the context developed. In contrast, efficient baselines such as Mamba 2, Gated DeltaNet, and SWA hit a ceiling, with their performance degrading or flattening out after 32,000 tokens.
The new TTT-E2E method was successfully adapted to context length, mimicking the behavior of Full Attention. In experiments using 3B parameter models, TTT-E2E actually maintained lower perplexity (better performance) than Full Attention throughout the pop-up window.
It is important to note that this performance did not come at the expense of speed. In terms of inference latency, TTT-E2E matched the efficiency of RNNs. With a context length of 128,000 tokens, TTT-E2E was 2.7 times faster than the Full-Attention Transformer on Nvidia H100 hardware.
Sun notes that, crucial for adoption, TTT models can be deployed today for inference on standard Transformer infrastructure to achieve these speedups. However, he cautions that the training side of the equation (especially the outer loop) is currently more complex and slower than standard methods, representing a hurdle that still requires technical optimization.
The benefits become even more significant as the data evolves. Sun says the advantage is expected to widen further in settings involving a million tokens, although these numbers are projections rather than current baseline deployments.
However, this approach has specific limitations related to its design philosophy. The researchers carried out a "A needle in a haystack" test, which requires the model to retrieve a specific, isolated piece of information (like a password) hidden in a large block of text. In this evaluation, Full Attention significantly outperformed all other methods, including TTT-E2E.
Indeed, Full Attention relies on a cache that allows almost lossless recall of specific details, while TTT-E2E relies on compression. Compression perfectly captures intuition and basic information, but can lose specific, random details that don’t fit the learned patterns.
This distinction has major implications for enterprise data pipelines, particularly RAG. Sun suggests that TTT will not make RAG obsolete but will redefine it. He compares TTT to "update the human brain" with general knowledge, while RAG will remain a necessary tool for precision, "in the same way that humans still need to write things down in a notepad." For enterprise teams, the takeaway is that TTT reduces how often you need recovery, but doesn’t eliminate the need for exact external memory.
Although the technique was demonstrated on the Transformer architecture, the researchers note that “in principle, TTT can be applied to any base architecture” allowing long-term and short-term memory components to be separated.
“We believe that these two classes of memory will continue to complement each other," conclude the researchers.
Looking ahead, Sun predicts a paradigm shift where the primary form of AI memory will be highly compressed rather than exact. Although the models will retain a "reasonable" perfect recall window of around 128,000 tokens, he believes TTT architectures will eventually unlock a "compressed memory of billions of tokens," fundamentally changing the way enterprise agents balance recall, cost, and context duration.
