TL;DR
This paper introduces a diagnostic framework that probes the inner workings of feed-forward 3D reconstruction models by systematically corrupting their internal representations. By stabilizing the coordinate-frame drift that occurs between different perturbation strengths, the authors reveal how low-level and high-level features uniquely shape 3D geometry. The framework is also showcased in DecArt, an interactive VR art installation that visualizes these degradation trajectories as a metaphor for "computational forgetting."
What Problem Does This Paper Address?
Modern feed-forward 3D reconstruction systems can instantly transform 2D images into explicit 3D spatial representations (like point clouds or Gaussian splats). They achieve this by blending low-level image processing (texture, edges) with high-level semantic features from pretrained visual encoders like DINOv2.
However, these models operate largely as black boxes. Simply evaluating the final output quality doesn't tell us how specific internal representations constrain the predicted 3D geometry, or how an upstream error might propagate through a scene. For embodied agents (robots) or XR systems that rely on accurate geometry to interact with the world, understanding these internal dependencies is crucial.
Methods
The authors develop a controlled intervention framework that perturbs a fixed reconstruction backbone at two distinct internal loci:
- Processor-Level (Low-Level): Modifies the parameters of a lightweight convolutional processor operating directly on images. Interventions include spectral attenuation (suppressing high-frequency kernel components via FFT), spatially correlated weight noise, and parameter sparsification (masking weights to zero).
- Semantic-Level (High-Level): Modifies DINOv2 patch features. It uses a direction-changing transformation (via QR-decomposition) to progressively misalign features from their expected basis while preserving structural signal, mixed with stochastic Gaussian noise.
Crucially, these semantic interventions can be globally applied or restricted to a specific object using a 2D mask generated by Grounded-SAM.
Overcoming Cross-Strength Drift
A major technical challenge is that running the reconstruction model independently at varying intervention strengths ($\lambda$) causes the model to re-estimate the global coordinate frame each time. This introduces an artificial "rigid drift" (jumping and shifting) that ruins visual comparisons.
To fix this, the authors introduce Canonical Background Anchoring. They isolate the non-intervened background, use point-to-plane Iterative Closest Point (ICP) to register the drifted backgrounds back to a canonical reference ($\lambda=0$), and apply that same transformation to the target object. Finally, they use Pixel-Indexed Interpolation to smoothly fill in intermediate states between sparse anchor points.
Datasets
The paper does not introduce or evaluate on standard multi-dataset quantitative benchmarks. Instead, the evidence is qualitative and hypothesis-generating, utilizing curated and user-provided image and video sequences (such as a toy train set and a "break dancer" sequence) to observe the effects of the interventions.
Key Insights and Ideas
- Locus-Dependent Failures: Low-level processor corruption primarily leads to a loss of crisp edges, unstable fine textures, and fragmented or translucent geometry. Semantic corruption, on the other hand, causes severe topological distortions—surfaces stretch, bend, or merge abnormally.
- Structured Misalignment vs. Information Loss: At moderate intervention strengths, semantic corruption doesn't just destroy the object into random noise. Instead, it produces "structured failures" where the geometry deforms drastically but still retains recognizable visual traces of the source object.
- Contextual Coupling: Even when an intervention is strictly masked to a single object, downstream feature fusion causes the geometric corruption to slightly bleed into neighboring background regions (creating "halos"), proving that object representations are contextually coupled within the network.
Key Results
The primary result is a validated diagnostic tool that successfully maps out continuous "reconstruction-response trajectories" as intervention strength increases.
The framework's practical utility is demonstrated through DecArt, an interactive screen- and VR-based installation. Users can manipulate a slider to alter $\lambda$ in real-time, watching a 3D scene degrade smoothly without jarring coordinate jumps, providing an intuitive look at how deep networks organize spatial data.
Limitations and Future Work
- Qualitative Scope: The findings are currently hypothesis-generating and limited to a single feed-forward reconstruction family. Broadening these insights into a universal theory will require multi-model quantitative benchmarks and geometric frequency metrics.
- ICP Limitations: The background anchoring step corrects for rigid coordinate drift but cannot fix local deformations or missing geometry caused by the interventions.
- Boundary Leakage: Downsampling the 2D segmentation masks to match the DINOv2 token grid introduces boundary uncertainties, contributing to the observed contextual leakage.
Summary updated