Sandwich-Residuals: Parameter-Efficient Test-time Adaptation of World Models

Krishnam Soni, Aditya Sehgal, Vedant Dave, Elmar RueckertarXiv 2026 · arXiv

TL;DR

When deploying robotic world models in new environments, standard test-time adaptation methods fine-tune millions of internal parameters, which is computationally heavy and can degrade pretrained representations. Sandwich-Residuals offers a radically simple alternative: freeze the entire pretrained world model and only train tiny residual blocks at its input and output interfaces. This approach uses 97–99% fewer trainable parameters while matching or exceeding the performance of full-model adaptation under severe appearance and dynamics shifts.

What Problem Does This Paper Address?

World models allow robots to plan actions by predicting how their environment will change. However, when a robot encounters a test-time environment that differs from its training data—due to changes in lighting, camera angles (appearance shifts), or friction and mass (dynamics shifts)—these predictions break down.

To fix this online, current Test-Time Adaptation (TTA) methods like AdaJEPA fine-tune the internal layers of the world model using self-supervised prediction errors. While effective, this strategy modifies millions of parameters and requires researchers to carefully choose which internal blocks to adapt for different tasks. The authors of this paper ask a fundamental question: Do we actually need to modify the internal weights of a pretrained world model to adapt it to new test-time conditions?

Methods

Instead of altering the core world model, the authors introduce Sandwich-Residuals (SR). They freeze all pretrained encoder and predictor parameters and "sandwich" the predictor between lightweight, learnable residual modules.

Fig. 1: Overview and mechanism of Sandwich Residuals (SR). Left: At each replanning step, the world model predicts the latent representation of the next observation under the selected action at, shown schematically as ˆot+1, while executing that action yields the observed next observation ot+1. As test-time adaptation progresses, updates from these observed transitions reduce the prediction mismatch, illustrating progressive online recalibration of the model. Right: Circled + symbols denote additive residual connections. The orange residual blocks are adapted during test-time adaptation, while the encoders and predictor remain frozen. See Section III for notation and variable definitions.

As shown in Figure 1, the architecture consists of two adaptation pathways:

  1. Input-side corrections: Small residual layers modify the visual and action embeddings before they enter the predictor. Crucially, the action residual is conditioned on the robot's current proprioception (internal state), allowing the same command to be corrected differently depending on the state.
  2. Output-side corrections: Residual modules correct the predicted visual and proprioceptive latents before they are passed to the planner or used for recursive rollouts.

During deployment, these residuals are optimized online using the model's own self-supervised latent prediction error over a tiny buffer of the 5 most recent transitions. It requires zero rewards, labels, or source-domain data.

Datasets

The method was evaluated across two distinct setups:

  • AdaJEPA Benchmark: Comprises 21 primary and 7 compound distribution shifts across four goal-conditioned visual control tasks: Medium Maze, Diverse Maze, PushT (pushing a T-shaped block), and PushObj (manipulating multiple shapes). Shifts included changes in simulator physics (damping, density), image corruptions (blur, noise, brightness), and unseen object shapes or layouts.
  • OGBench-Cube: A 3D robotic manipulation task using a UR5e arm with a parallel gripper. This setup utilized a completely different architecture—a DINO-WM world model featuring a frozen DINOv2 vision backbone and a Vision Transformer (ViT) predictor, evaluated under lighting, viewpoint, and dynamics shifts.

Key Insights and Ideas

  • Interface-level adaptation is enough: You don't need to change how a model "thinks" internally to fix its predictions. Adjusting the inputs and outputs (the interfaces) is sufficient for planning.
  • Pathway Specialization: An ablation study revealed that different types of shifts rely on different parts of the sandwich. Output-side corrections dominate in maze environments (handling appearance and layout changes), whereas input-side action corrections are vital when the physics change (like controller gains in the pushing task).
  • Preserving Latent Geometry: By keeping the core model frozen, the method avoids the risk of "catastrophic forgetting" or degrading the robust representations learned during large-scale pretraining.

Key Results

  • Massive Parameter Savings: On the AdaJEPA benchmark, Sandwich-Residuals adapted only ~100k parameters—a 97% to 99% reduction compared to AdaJEPA variants which update between 3.6M and 9.9M parameters.
  • High Success Rates: Across all 21 primary conditions, the method achieved 1.3× the success rate of the frozen baseline, retaining 95% of the performance of the best full-adaptation method.
  • Superior Under Compound Shifts: When multiple corruptions were applied simultaneously (e.g., image blur combined with 50× simulator damping), Sandwich-Residuals achieved 1.9× the success rate of the frozen model, outperforming internal block adaptation in several scenarios.
Fig. 2: Example episodes under compound shifts, planned with each of the four models, from the same start state and goal: (a) Medium Maze, blur σ = 2 + damping 50×; (b) PushT, blur σ = 2 + controller kv 2×; (c) PushObj T, blur σ = 2 + controller kv 2×; (d) PushObj square, blur σ = 2 + controller kv 2×. Dotted green lines trace the executed trajectory. Frozen fails in all four episodes; ours succeeds in all four, uniquely reaching the goal under 50× damping in (a), while predfirst fails in (c) and predlast fails in (d). These are illustrative single episodes and not representative of the aggregate rates in Tables I and II.
  • Cross-Architecture Transfer: When applied to the DINO-WM model on 3D manipulation, Sandwich-Residuals was the only adaptation strategy that improved performance over the frozen baseline across all test-time shifts (lighting, viewpoint, and arm gain).

Limitations and Future Work

While highly parameter-efficient, the authors note that reducing trainable parameters does not automatically result in a proportional reduction in computational cost, because gradients must still be backpropagated through the frozen predictor layers to update the input-side residuals.

Additionally, the method assumes that the pretrained visual encoder retains enough information to be useful; if a severe shift completely destroys features during encoding, downstream residuals cannot recover them. The authors suggest that future work could introduce lightweight corrections earlier in the visual pipeline or explore retaining these residuals across multiple episodes for persistent real-world deployment.

Summary updated

Abstract

Latent world models enable planning by predicting the effects of actions in a learned representation space, but their predictions can become unreliable when test-time conditions differ from training. Existing test-time adaptation methods address this by updating parts of the pretrained model, often modifying millions of parameters and requiring a choice of which internal components to adapt. We introduce Sandwich-Residuals, a lightweight alternative that keeps the pretrained world model frozen and learns only small residual corrections around the predictor. The residuals are optimized online using the model's self-supervised prediction error and require no rewards, labels, or source-domain data. Across 21 conditions on the AdaJEPA benchmark, our method achieves $1.3\times$ the success rate of the frozen model while retaining 95% of the performance of the strongest AdaJEPA variant and adapting 97-99% fewer parameters. Under compound shifts, this advantage increases to $1.9\times$ the success rate of the frozen model, while remaining comparable to internal block adaptation. We further demonstrate the same adaptation principle on a DINO-WM model for 3-D manipulation. These results suggest that effective test-time adaptation of world models does not necessarily require modifying their pretrained internal weights.
BibTeX
@misc{soni2026sandwich,
  author = {Krishnam Soni and Aditya Sehgal and Vedant Dave and Elmar Rueckert},
  title = {{Sandwich-Residuals: Parameter-Efficient Test-time Adaptation of World Models}},
  year = {2026},
  eprint = {2609.21740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.21740},
}
Figure