TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models

Chin-Yang Lin, Ryo Hachiuma, Jaesung Choe, Min-Hung Chen, Yu-Chiang Frank Wang, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu, Cheng SunarXiv 2026 · arXiv

TL;DR

Test-time Baking (TTB) bridges the gap between high-quality but slow attention-based view synthesis and fast but lower-quality linear-attention models. By "baking" global scene information from cross-view attention into lightweight, scene-specific MLPs, TTB achieves a 23x rendering speedup over state-of-the-art baselines while maintaining comparable image quality.

What Problem Does This Paper Address?

Large View Synthesis Models (LVSMs) have revolutionized our ability to generate novel viewpoints from a few images without needing explicit 3D geometry. However, they face a frustrating trade-off:

  1. High Quality, Low Speed: Decoder-only models (like DVSM) use cross-view attention to gather scene information. While accurate, the rendering time grows linearly with the number of input views, making real-time use impossible for complex scenes.
  2. High Speed, Low Quality: Recent "Test-Time Training" (TTT) approaches replace slow attention with fast-weight MLPs. While this provides constant-time rendering, it significantly sacrifices image detail, often resulting in blurry outputs.

The authors identify two main culprits for this quality loss: the MLP is forced to replace attention entirely rather than collaborate with it, and there is a "modality misalignment" where the MLP is trained on image-rich data but queried with camera-only data during rendering.

Methods

The core idea of TTB is to bake the knowledge of a powerful attention mechanism into a fast MLP during a one-time "prefilling" stage.

  • Collaborative Baking: Instead of using the MLP to predict raw pixel values (as in TTT), TTB repurposes it to explicitly learn the mapping of the cross-view attention layer. The MLP is trained to match the output of the attention mechanism for each scene.
  • Modality Alignment via Context Pairs: To fix the training-rendering mismatch, TTB disentangles context views into pairs: a "camera-only" rendering token (the input) and an "image-content" ground-truth token (the target). This ensures the MLP learns to retrieve scene information using the same type of camera-only queries it will encounter at test time.
  • Inner-loop Optimization: During the prefilling stage, the model performs a few steps of optimization (using the Muon optimizer and MSE loss) to adapt the MLP weights to the specific scene.
  • Local-Duplicate Inference: To prevent a single MLP from becoming "overloaded" with too much scene information, TTB clusters context views spatially and bakes separate MLPs for different regions of the scene.
The TTB pipeline showing how scene information is baked into MLPs during reconstruction to enable fast O(1) rendering.

Datasets

The authors evaluated TTB across several challenging benchmarks:

  • DL3DV: A large-scale dataset of real-world indoor and outdoor scenes, used as the primary benchmark for dense-view synthesis.
  • Re10K: Used to test performance in sparse, two-view input scenarios.
  • MipNeRF360, Free, and Hike: Used for "zero-shot" evaluation to prove the model generalizes to "in-the-wild" scenes it hasn't seen during training.

Key Insights and Ideas

  • Baking is Better than Replacing: The "aha moment" is that we don't have to choose between attention and MLPs. By using attention as a "teacher" during a pre-rendering phase, we can transfer its global reasoning capabilities into a much faster structure.
  • Solving the Modality Gap: Figure 2 illustrates how standard TTT methods suffer because they are trained on tokens that "know" the image content but are tested on novel-view tokens that only "know" the camera pose. TTB's context-pairing strategy aligns these distributions from the very first layer.
  • Constant-Time Rendering: Because the baked MLP has a fixed size regardless of how many input views were used, the rendering speed remains constant ($O(1)$ complexity per token), unlike standard attention which slows down as you add more context.
Motivation comparing TTT and TTB. Only TTB (c) aligns the data modalities between training and rendering.

Key Results

TTB effectively defines a new Pareto frontier for view synthesis, offering the best balance of speed and quality currently available.

  • Speed: It achieves up to 23x faster rendering than the DVSM baseline and is roughly 2.3x faster than the previous TTT-based state-of-the-art (LaCT).
  • Quality: TTB maintains a PSNR nearly identical to full-attention models. Qualitatively, it preserves high-frequency details like handrails, text, and fine textures that other fast methods blur out.
  • Efficiency: On the DL3DV benchmark, TTB delivers ~23 FPS at 960p resolution, whereas the high-quality DVSM baseline crawls at 1-4 FPS.
TTB sits on the Pareto frontier, providing significantly higher FPS than DVSM at comparable PSNR.
Qualitative comparison showing TTB preserves fine details (like the text on the fire hydrant) that other models blur.

Limitations and Future Work

  • Reconstruction Overhead: While rendering is lightning-fast, the "baking" process itself takes time (quadratic relative to the number of input views). This makes it less suitable for "instant" single-view applications but ideal for interactive walkthroughs of a pre-processed scene.
  • Capacity Limits: A single MLP has a fixed capacity. While "local-duplicate inference" helps, very large or complex scenes might still hit a quality ceiling.
  • Interpolation Focus: Like most LVSMs, TTB is designed for interpolating between views. It may struggle with extreme extrapolation into completely unobserved areas of a scene.

The authors suggest that this "baking" perspective could be applied beyond view synthesis—for example, to speed up long-context Large Language Models (LLMs) by baking their KV-caches into fast-weight states.

Summary updated

Abstract

Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to **23x** rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io
BibTeX
@misc{lin2026ttb,
  author = {Chin-Yang Lin and Ryo Hachiuma and Jaesung Choe and Min-Hung Chen and Yu-Chiang Frank Wang and Yen-Yu Lin and Wei-Chen Chiu and Yu-Lun Liu and Cheng Sun},
  title = {{TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models}},
  year = {2026},
  eprint = {2611.17331},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17331},
}
Figure