RoboTTT: Context Scaling for Robot Policies

Authors: Yunfan Jiang, Yevgen Chebotar, Ruijie Zheng, Fengyuan Hu, Yunhao Ge, Jimmy Wu, Tianyuan Dai, Scott Reed, Li Fei-Fei, Yuke Zhu, Linxi "Jim" Fan · CoRL 2026 2026

Why it matters

Today's robot foundation models, including GR00T, pi0 and most other VLAs, decide each action from the current camera frames and at most a few past ones. They cannot tell two similar-looking stages of a long task apart, remember where a part went when it is hidden, or learn from what just went wrong. RoboTTT shows that a policy can carry about five minutes of its own history at no extra cost per action, and that this memory pays off on real hardware: a two-arm robot follows a single human video to build a circuit it was never trained on, retries a missed screw instead of moving on, and puts back a part a person removes. The paper also reports that performance keeps rising as the pretraining context grows from 128 timesteps to 8K, which suggests context length as a scaling axis alongside data and model size. The method is a set of layers added to an existing model, so it could in principle be fitted to other open VLAs. The paper is an award finalist and an oral at CoRL 2026. The authors have not released code or weights, so for now the value to a builder is the recipe and the evidence, not a model to download.

Problem

Most state-of-the-art robot foundation models act on the current observation alone or on a short window of 2 to 8 past frames. Long manipulation tasks need more: visually similar stages have to be told apart, objects that are now hidden have to be remembered, and a policy that has failed should use that failure on its next attempt. Simply feeding past frames back in tends to hurt, because the policy latches onto spurious patterns in its own history. Transformers that attend over the whole history slow down as it grows, even with a key-value cache, and classic recurrent networks have too little capacity to compress thousands of timesteps of dense, repetitive robot data. No robot policy had shown closed-loop gains from a context of minutes.

Proposed solution

RoboTTT adds a test-time training (TTT) layer after the attention layers in each of the 16 layers of GR00T N1.7's diffusion transformer action head. Attention still works within a single timestep; the TTT layers work across time. Each TTT layer holds a small two-layer MLP whose weights, the fast weights, take one gradient update on every new observation, writing the history into the weights rather than into a growing cache, so the cost per action stays constant. A learned gate, started near zero, blends the TTT output into the pretrained model so its skills are kept. The model is trained on whole trajectories, with each action chunk given its own noise level (sequence action forcing), and with truncated backpropagation through time so that memory use does not grow with context length. Because some timesteps can update the fast weights without being imitation targets, the same model learns from a human video placed in its context, and, with DAgger Distillation, from rollouts in which its own mistakes are context and a human's corrections are the targets.

Key insights

Gradient-updated fast weights make a better memory for robot streams than a linear recurrent state: with the same placement and parameter count, Gated DeltaNet layers gain nothing from longer pretraining context, while RoboTTT keeps improving
Naively adding past frames hurts: GR00T N1.7 with one history frame scores 39.5% on the Pup Go Car task against 57% without it, because it picks up spurious patterns from its history
Each action chunk in a training sequence needs its own noise level; with one shared level, sequence training is unstable and the robot makes no meaningful progress
A policy's failed actions are useless as imitation targets but valuable as context: training GR00T N1.7 on full DAgger rollouts scores the same as on the corrections alone, while DAgger Distillation, which uses the failures only to update the fast weights, gains the most
The trained context should cover the episode: pretrained below 1K timesteps, about half a minute and shorter than any task episode, RoboTTT still beats Gated DeltaNet but falls short of its longer-context versions, which the authors put down to rollouts running past the window seen in training

Main results

Average task completion over three real bimanual assembly tasks is 79%, against 42% for GR00T N1.7 (87% higher), 49% with one history frame and 56% for the Gated DeltaNet variant (41% higher than that best baseline)
Fully successful trials: 9 of 20 on Pup Go Car, 13 of 20 on Circuit and 2 of 10 on the five-minute, ten-stage Gear Bot, the only full Gear Bot runs by any method; no baseline managed more than 3, 8 and 0
Closed-loop performance rises with pretraining context, to 71.5% at 8K timesteps against 43.9% at 1K (63% higher) and 45.6% for the best short-context baseline, with no sign of saturation
One-shot imitation from a single human video of an unseen Circuit configuration succeeds in 6 of 10 trials with a 65% completion score; the Gated DeltaNet variant succeeds in none
When a person removes a part the robot has just installed, RoboTTT reinstalls it in 15 of 20 roof trials and 18 of 20 tire trials, against at most 10 of 20 and 11 of 20 for the short-context GR00T baselines
From a pool of 100 DAgger rollouts, DAgger Distillation improves RoboTTT by 36%; ordinary DAgger fine-tuning improves the four methods by 9% on average
In ablations, a linear fast model scores 27% below the MLP, passing action tokens through the TTT layers adds 23% and learned register tokens add a further 18%; register tokens alone do not help GR00T N1.7

Limitations

No code or weights from the authors: as of 8 November 2026 the arXiv record and the project page link no repository, and there is none in NVIDIA's GitHub organisations, in the Isaac-GR00T repository or among NVIDIA's Hugging Face models; an unofficial MIT-licensed re-implementation of the layer (lucidrains/robo_ttt) is not from the authors and is a layer library, not a trained policy
The paper names its own limits: longer training context costs more compute, the TTT layers use a generic key-value loss rather than an objective designed for robots, and the policy does not fix every failure; the authors suggest adding reinforcement learning
All results come from one robot setup, a YAM bimanual station with four cameras, on three toy assembly tasks in the authors' lab, with 10 to 20 trials per method and no confidence intervals
The method is shown only on GR00T N1.7; the claim that it fits other robot foundation models is not tested
Reproducing it needs data-centre hardware: pretraining ran on 16 NVIDIA GB200 GPUs on a robot and human-video mixture that has not been released; only inference fits on a workstation with an RTX 5090
Scores use the authors' own task rubrics, and the abstract gives the 8K-over-1K gain as 62% while the body and the reported scores give 63%

Field impact

Context length has been one of the main ways language models scale, but robot policies have stayed at one frame or a handful. The authors report, for the first time, steady closed-loop gains on real robots as pretraining context grows to minutes, which makes context length something robot model builders can now measure and trade off against data and parameters. It also brings test-time training, until now shown mainly in language, video and 3D vision, into robot control, and turns two familiar data sources into context: human demonstration videos for one-shot imitation, and DAgger interventions, where the robot's own failures become the signal for on-the-fly correction. A comparable recurrent layer without gradient updates (Gated DeltaNet) did not show the same scaling, so the result points at the update rule, not just at having memory.

Abstract

Recent robot foundation models operate with single-step or short-history visuomotor context. We introduce Test-Time-Training Robot Policies (RoboTTT), a robot model and training recipe that scale visuomotor context to 8K timesteps, three orders of magnitude beyond state-of-the-art policies, without growing inference latency. At this context length, we unlock new robot capabilities: one-shot in-context imitation from human video demonstrations, on-the-fly policy improvement, robustness to perturbations, and stronger performance on multi-stage, long-horizon tasks. We also observe, for the first time, steady gains in closed-loop performance as pretraining context length scales. At its core, RoboTTT integrates Test-Time Training into robot foundation models such as Vision-Language-Action policies, yielding a sequence model whose recurrent state consists of fast weights, parameters updated by gradient descent during both training and inference, compressing histories into weight space and retrieving contextual information for long-context conditioning. To scale training context length, the recipe combines sequence action forcing with truncated backpropagation through time. On challenging real-robot manipulation tasks, RoboTTT improves overall performance by 87% over the single-step context baseline and fully completes a five-minute, ten-stage assembly task, which no baseline ever does. RoboTTT trained with 8K-timestep context outperforms the same model pretrained with 1K timesteps by 62%, suggesting context length as a new scaling axis for robot foundation models. Videos are available at research.nvidia.com/labs/gear/robottt.

Open Source Robotics