TL;DR
Test-time Baking (TTB) bridges the gap between high-quality but slow attention-based view synthesis and fast but lower-quality linear-attention models. By "baking" global scene information from cross-view attention into lightweight, scene-specific MLPs, TTB achieves a 23x rendering speedup over state-of-the-art baselines while maintaining comparable image quality.
What Problem Does This Paper Address?
Large View Synthesis Models (LVSMs) have revolutionized our ability to generate novel viewpoints from a few images without needing explicit 3D geometry. However, they face a frustrating trade-off:
- High Quality, Low Speed: Decoder-only models (like DVSM) use cross-view attention to gather scene information. While accurate, the rendering time grows linearly with the number of input views, making real-time use impossible for complex scenes.
- High Speed, Low Quality: Recent "Test-Time Training" (TTT) approaches replace slow attention with fast-weight MLPs. While this provides constant-time rendering, it significantly sacrifices image detail, often resulting in blurry outputs.
The authors identify two main culprits for this quality loss: the MLP is forced to replace attention entirely rather than collaborate with it, and there is a "modality misalignment" where the MLP is trained on image-rich data but queried with camera-only data during rendering.
Methods
The core idea of TTB is to bake the knowledge of a powerful attention mechanism into a fast MLP during a one-time "prefilling" stage.
- Collaborative Baking: Instead of using the MLP to predict raw pixel values (as in TTT), TTB repurposes it to explicitly learn the mapping of the cross-view attention layer. The MLP is trained to match the output of the attention mechanism for each scene.
- Modality Alignment via Context Pairs: To fix the training-rendering mismatch, TTB disentangles context views into pairs: a "camera-only" rendering token (the input) and an "image-content" ground-truth token (the target). This ensures the MLP learns to retrieve scene information using the same type of camera-only queries it will encounter at test time.
- Inner-loop Optimization: During the prefilling stage, the model performs a few steps of optimization (using the Muon optimizer and MSE loss) to adapt the MLP weights to the specific scene.
- Local-Duplicate Inference: To prevent a single MLP from becoming "overloaded" with too much scene information, TTB clusters context views spatially and bakes separate MLPs for different regions of the scene.
Datasets
The authors evaluated TTB across several challenging benchmarks:
- DL3DV: A large-scale dataset of real-world indoor and outdoor scenes, used as the primary benchmark for dense-view synthesis.
- Re10K: Used to test performance in sparse, two-view input scenarios.
- MipNeRF360, Free, and Hike: Used for "zero-shot" evaluation to prove the model generalizes to "in-the-wild" scenes it hasn't seen during training.
Key Insights and Ideas
- Baking is Better than Replacing: The "aha moment" is that we don't have to choose between attention and MLPs. By using attention as a "teacher" during a pre-rendering phase, we can transfer its global reasoning capabilities into a much faster structure.
- Solving the Modality Gap: Figure 2 illustrates how standard TTT methods suffer because they are trained on tokens that "know" the image content but are tested on novel-view tokens that only "know" the camera pose. TTB's context-pairing strategy aligns these distributions from the very first layer.
- Constant-Time Rendering: Because the baked MLP has a fixed size regardless of how many input views were used, the rendering speed remains constant ($O(1)$ complexity per token), unlike standard attention which slows down as you add more context.
Key Results
TTB effectively defines a new Pareto frontier for view synthesis, offering the best balance of speed and quality currently available.
- Speed: It achieves up to 23x faster rendering than the DVSM baseline and is roughly 2.3x faster than the previous TTT-based state-of-the-art (LaCT).
- Quality: TTB maintains a PSNR nearly identical to full-attention models. Qualitatively, it preserves high-frequency details like handrails, text, and fine textures that other fast methods blur out.
- Efficiency: On the DL3DV benchmark, TTB delivers ~23 FPS at 960p resolution, whereas the high-quality DVSM baseline crawls at 1-4 FPS.
Limitations and Future Work
- Reconstruction Overhead: While rendering is lightning-fast, the "baking" process itself takes time (quadratic relative to the number of input views). This makes it less suitable for "instant" single-view applications but ideal for interactive walkthroughs of a pre-processed scene.
- Capacity Limits: A single MLP has a fixed capacity. While "local-duplicate inference" helps, very large or complex scenes might still hit a quality ceiling.
- Interpolation Focus: Like most LVSMs, TTB is designed for interpolating between views. It may struggle with extreme extrapolation into completely unobserved areas of a scene.
The authors suggest that this "baking" perspective could be applied beyond view synthesis—for example, to speed up long-context Large Language Models (LLMs) by baking their KV-caches into fast-weight states.
Summary updated