Reconstructing a 3D scene from two or three photos now often works in two stages. A diffusion model, which generates images by gradually denoising random noise, first imagines dozens of extra views of the scene. A 3D model is then fitted to those views. The first stage is slow because every imagined view needs many denoising passes through a large network. The usual speed-up is to cache: skip the network on some steps and reuse features from an earlier step. Current caches make this skip-or-compute decision the same way for every part of every image. That choice is wasteful. It also has a less obvious cost in 3D. Caching errors that differ from one generated view to the next look like disagreements about geometry, and the 3D fit turns disagreements into floaters, the semi-transparent debris that sparse-view methods already struggle with.
We propose to decide caching per image token (one small patch of an image), using a visibility test borrowed from depth-aware gaze-target detection. Some tokens in an imagined view land on surfaces that the real input photos already see. Their content is mostly fixed by the conditioning, so they can be cached aggressively. Tokens that land on surfaces no input photo sees are where the model is actually inventing content, and they get full compute. Tokens in different views that hit the same 3D surface point are refreshed on the same denoising steps, so any drift from caching is shared across views instead of contradicting between them. The claim is that, at equal wall-clock budget, this gives better downstream 3D reconstructions, measured on held-out real views and by floater counts, than caching schemes that ignore geometry. The claim is not that it gives better-looking individual images.
Where this sits
The pipeline we accelerate is the one from CAT3D. CAT3D adapts a video diffusion model by replacing time embeddings with "raymaps" (per-pixel camera ray origins and directions). It generates 80 to 960 consistent views from one to three inputs in about a minute, then fits a modified Zip-NeRF. CAT3D motivates this work in two ways. Its generated views are only approximately consistent across viewpoints, and the leftover inconsistency shows up as floaters and blur. Its minute-scale latency is the cost we want to cut.
The caching methods supply our machinery.
- BudCache chooses a fixed compute-versus-cache mask offline, by simulated annealing followed by hill climbing. It then tunes the timestep schedule to correct the drift that caching causes.
- DiCache runs a one-layer probe to decide online whether to skip the deeper layers. It extrapolates cached features to first order along the probe's trajectory, rather than copying them unchanged.
- Token Caching for Diffusion Transformer Acceleration shows that caching can be done per token rather than per step.
All three judge redundancy from image or feature statistics alone. None uses the scene's geometry, and none evaluates in 3D. Our contribution is the geometric gate and the rule that synchronizes refreshes across views. We reuse DiCache's first-order extrapolation for the tokens we cache.
The visibility test comes from gaze-target detection. ESCNet builds a metric point cloud from monocular depth and keeps only the "front-most" surface along rays cast from the head, since gaze cannot pass through solid objects. Gaze-cone methods such as Dual Attention Guided Gaze Target Detection in the Wild intersect a cone with a depth map, which acts as a depth band-pass filter. We change what the test is used for: the ray starts at each imagined camera instead of at a person's eye, and its output is a compute allocation instead of an attention target.
The same literature also warns about the gate's weak point. Monocular depth has unknown scale and shift and bleeds across object boundaries. Kumar and Vats's depth-aware 3DGS handles the scale problem with a scale-invariant depth loss. We use the same scale-and-shift alignment to bring per-view depth into one shared frame.
Core claim and mechanism
We first build a coarse proxy geometry for the scene. The default is to run monocular depth on each input photo, align the depths to one another by fitting scale and shift in the scale-invariant sense, and fuse the result into a point set. The alternative is a single forward pass of MVSplat.
For each latent token of each imagined view, we cast the token's ray through the raymap and find its front-most intersection with this proxy. We then check whether that point is visible from some input camera, within an angular tolerance and a depth-consistency band. The band plays the role of the gaze cone's tolerance.
This sorts tokens into three groups:
- Observed: the front-most point is visible and its depth estimate is confident.
- Unobserved: the point is occluded in, or lies outside, every input view.
- Uncertain: the depth estimate is low-confidence, for example near boundaries or where the monocular estimators disagree after alignment. These are treated as unobserved.
Given a budget of network evaluations averaged over tokens, compute is allocated as follows. Unobserved tokens are recomputed on every step. Observed tokens are recomputed only at a sparse set of steps chosen by a BudCache-style offline search, and are extrapolated DiCache-style in between. Tokens in different views whose front-most points fall within a small 3D radius of one another are grouped, and each group is refreshed on the same steps. Cached tokens still supply stale keys and values to the attention layers, so the full-compute tokens continue to condition on them.
The hypothesis behind synchronized refresh can be stated as a testable prediction. When two views are generated from the same stale features and extrapolated the same way, they drift in the same direction. A shared drift leaves the 3D fit with a slightly shifted but consistent surface. Independent drifts give it contradictory evidence, which the fit resolves by producing floaters.
Research program
The work has three parts.
Build and measure the gate. We implement the visibility gate and the synchronized token cache on a raymap-conditioned multi-view diffusion model. If a CAT3D-class model is not available to us, we fine-tune an open video model on raymaps following CAT3D's recipe. The model's quality is not the object of study. Only relative comparisons between caching schemes on the same model matter.
Test whether image-level metrics predict 3D damage. For every caching scheme we record per-view similarity to the uncached output. We also record a cross-view reprojection error: warp each generated view into its neighbours using the dense reconstruction's depth and measure disagreement. Finally we record the quality of the downstream Zip-NeRF or 3DGS fit. We expect the ranking of caching schemes by per-view LPIPS to disagree with their ranking by 3D quality, and cross-view reprojection error to track 3D quality better. If that holds, it is a finding in its own right for the caching literature.
Treat synthetic-to-real transfer as a constraint on the gate. The offline schedule search and the gate's tolerances are calibrated on rendered synthetic scenes, where depth is exact, and then applied to real captures, where depth is monocular and flawed. The literature on the synthetic-to-real gap predicts a shortcut here. ASM shows that clean synthetic inputs let a model latch onto the absence of real sensor artifacts. A gate tuned on perfect depth may mark as "observed" many regions that real depth gets wrong. Following ASM's idea of corrupting synthetic data until it resembles real data, we calibrate on synthetic depth corrupted with dropout, random affine scale-shift error, and boundary bleeding. We then compare the synthetic-to-real transfer of the resulting schedule against a schedule calibrated directly on a small real calibration set.
Baselines
The uncached reference runs the multi-view diffusion model for its full step count. It sets the quality ceiling and the latency to beat. Every other method is reported as a speedup relative to it and as a gap to its 3D metrics.
The naive fast baseline simply uses fewer sampling steps at the same wall-clock budget, with nothing cached. Any caching scheme that fails to beat it is not worth its complexity.
The strongest existing competitors are DiCache and BudCache, applied to the same model with the same budgets in network evaluations and in wall-clock time. On the video model closest to our setting, DiCache reports a 2.45x speedup on Wan 2.1 at LPIPS 0.1734 (DiCache). BudCache reports cutting Wan2.1 latency from 189 s to 82 s at reconstruction PSNR 25.93, against 21.52 for TeaCache (BudCache). These are 2D video-fidelity numbers. No reported result was found on Lacuna for any caching method evaluated by downstream 3D reconstruction quality.
For BudCache, the file notes that schedule alignment requires backpropagating through sampling, which becomes prohibitive on multi-view volumes. We therefore run its mask search at the same search budget we give our own search, and include schedule alignment only if it fits in memory. We report which variant was run. Beating these baselines means lower 3D error on held-out real views at matched wall-clock time, or matched 3D error at a speedup that is at least 20% higher, across both view-count settings.
The key ablations isolate each ingredient on the same token-level cache:
- Replace the visibility gate with DiCache-style feature-difference selection, which keeps token granularity but removes geometry.
- Replace it with a random gate that marks the same fraction of tokens as cached.
- Remove cross-view synchronization while keeping the gate.
- Use the uncorrupted synthetic calibration instead of the corrupted one.
The reconstruction-side references are MVSplat as a fast feed-forward method with no diffusion, and Kumar and Vats's depth-aware 3DGS as a per-scene optimizer with no generative prior. No reported LLFF metric value for either was found on Lacuna in the file, so both will be run ourselves.
Why not just…?
Why not just distill the multi-view model to a few steps? Distribution-matching distillation, including the diversity-preserving variant in Diversity-Preserved Distribution Matching Distillation, can give larger speedups than caching. This is a real threat. Our answer is partial. Distillation needs a separate expensive training run for each model. It is known to reduce sample diversity, which in the unobserved regions is exactly the content we need. Nothing in the file shows that distilled multi-view models stay consistent across views. The two approaches can also be stacked: a gate can decide which tokens of a four-step student to cache. If budget permits, we will distill a four-step student and compare its 3D quality with our cache at matched latency. If the student wins clearly, the geometric-gate idea should move onto distilled models rather than be defended as an alternative to them.
Why not just use existing caching and let the reconstruction absorb the errors? CAT3D already down-weights generated views and uses a perceptual loss when fitting. One could argue this makes 3D-aware caching unnecessary. The file points the other way: CAT3D's residual cross-view inconsistency already produces floaters before any caching is added. The decisive test is direct. At matched budget, we measure whether DiCache and BudCache increase floaters and held-out error relative to the uncached model more than our method does, when both are fitted with CAT3D's robust weighting.
Why not skip diffusion and use a feed-forward reconstructor? MVSplat runs in a single forward pass and places density only where views correspond. For narrow baselines with little disoccluded area, it may simply win. We will stratify scenes by the fraction of target-view tokens that the gate marks unobserved. We expect MVSplat to be competitive when that fraction is low and to fall behind the diffusion pipeline when it is high. If MVSplat matches our method across all strata, the pipeline is unjustified at these view counts, and we would say so.
Why not learn token importance from features instead of from geometry? Feature-difference token selection needs no depth at all, so it cannot be misled by bad monocular depth. The ablation that swaps our gate for this selector settles the question. If it matches our 3D metrics, the geometry adds nothing and the contribution reduces to cross-view synchronization, which we test separately.
Proposed experiments and decisive checks
Real-scene evaluation uses NeRF LLFF with three and five input views, the setting of Kumar and Vats, and the multi-view setups of EvalMVX. We report PSNR, SSIM, and LPIPS on held-out real views, the cross-view reprojection error of the generated views, and a floater measure. The floater measure is the opacity mass of the fitted model lying in free space, as defined by a reconstruction from all available views. We also report end-to-end wall-clock time, including the cost of building the proxy geometry and running the gate. That overhead has to be counted, just as DiCache's probe costs about 4% of runtime. Synthetic calibration scenes are rendered with exact depth, and the real calibration set, used only for the transfer comparison, is a small held-out split of the real data.
Three checks decide the proposal.
- The core claim. At matched wall-clock time, visibility-gated synchronized caching must beat DiCache and BudCache on held-out 3D error and floater mass. If it only matches them, the idea fails.
- The mechanism. Removing synchronization must increase cross-view reprojection error and floaters while leaving per-view LPIPS roughly unchanged. That pattern would show that 3D damage from caching is driven by inconsistency between views rather than by per-view quality.
- The synthetic-to-real constraint. The gap between the corrupted-synthetic calibration and the real calibration must be smaller than the gap between the clean-synthetic calibration and the real calibration. If clean-synthetic calibration transfers equally well, the shortcut worry was unfounded and the corruption step can be dropped.
We do not expect to reach the sub-50 ms latency of interactive perception. The goal is a better trade-off between speed and 3D quality on the same model, and an evaluation standard for caching that measures what reconstruction actually needs.
References
- CAT3D: Create Anything in 3D with Multi-View Diffusion Models
- Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields
- Budget-Constrained Step-Level Diffusion Caching
- DiCache: Let Diffusion Model Determine Its Own Cache
- Token Caching for Diffusion Transformer Acceleration
- ESCNet: Gaze Target Detection with the Understanding of 3D Scenes
- Dual Attention Guided Gaze Target Detection in the Wild
- Few-Shot Novel View Synthesis Using Depth Aware 3D Gaussian Splatting
- MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-view Images
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesis
- EvalMVX: A Unified Benchmarking for Neural 3D Reconstruction under Diverse Multiview Setups