Probing Spatial Representations in Feed-Forward 3D Reconstruction through Controlled Intervention

Matti Marino Schlenker, Fran Zezelj, Philipp M. H. Bernhardt, Sophie Anderle, Polina Karpikova, Gege GaoODEWM @ ECCV 2026 · OpenReview

TL;DR

This paper introduces a diagnostic framework that probes the inner workings of feed-forward 3D reconstruction models by systematically corrupting their internal representations. By stabilizing the coordinate-frame drift that occurs between different perturbation strengths, the authors reveal how low-level and high-level features uniquely shape 3D geometry. The framework is also showcased in DecArt, an interactive VR art installation that visualizes these degradation trajectories as a metaphor for "computational forgetting."

What Problem Does This Paper Address?

Modern feed-forward 3D reconstruction systems can instantly transform 2D images into explicit 3D spatial representations (like point clouds or Gaussian splats). They achieve this by blending low-level image processing (texture, edges) with high-level semantic features from pretrained visual encoders like DINOv2.

However, these models operate largely as black boxes. Simply evaluating the final output quality doesn't tell us how specific internal representations constrain the predicted 3D geometry, or how an upstream error might propagate through a scene. For embodied agents (robots) or XR systems that rely on accurate geometry to interact with the world, understanding these internal dependencies is crucial.

Methods

The authors develop a controlled intervention framework that perturbs a fixed reconstruction backbone at two distinct internal loci:

  1. Processor-Level (Low-Level): Modifies the parameters of a lightweight convolutional processor operating directly on images. Interventions include spectral attenuation (suppressing high-frequency kernel components via FFT), spatially correlated weight noise, and parameter sparsification (masking weights to zero).
  2. Semantic-Level (High-Level): Modifies DINOv2 patch features. It uses a direction-changing transformation (via QR-decomposition) to progressively misalign features from their expected basis while preserving structural signal, mixed with stochastic Gaussian noise.

Crucially, these semantic interventions can be globally applied or restricted to a specific object using a 2D mask generated by Grounded-SAM.

Fig. 2: Intervention sites in a feed-forward spatial reconstruction pipeline. A shallow CNN processes local image evidence, while DINOv2 produces semantic patch features. A strength parameter λ controls processor-level or semantic-feature corruption before both streams are fused by the fixed reconstruction backbone into dense 3D geometry. A prompt-derived token mask optionally restricts semantic intervention to a selected object, enabling its localized geometric response to be examined.

Overcoming Cross-Strength Drift

A major technical challenge is that running the reconstruction model independently at varying intervention strengths ($\lambda$) causes the model to re-estimate the global coordinate frame each time. This introduces an artificial "rigid drift" (jumping and shifting) that ruins visual comparisons.

To fix this, the authors introduce Canonical Background Anchoring. They isolate the non-intervened background, use point-to-plane Iterative Closest Point (ICP) to register the drifted backgrounds back to a canonical reference ($\lambda=0$), and apply that same transformation to the target object. Finally, they use Pixel-Indexed Interpolation to smoothly fill in intermediate states between sparse anchor points.

The point cloud stabilization pipeline using ICP on the background to eliminate coordinate drift.

Datasets

The paper does not introduce or evaluate on standard multi-dataset quantitative benchmarks. Instead, the evidence is qualitative and hypothesis-generating, utilizing curated and user-provided image and video sequences (such as a toy train set and a "break dancer" sequence) to observe the effects of the interventions.

Key Insights and Ideas

  • Locus-Dependent Failures: Low-level processor corruption primarily leads to a loss of crisp edges, unstable fine textures, and fragmented or translucent geometry. Semantic corruption, on the other hand, causes severe topological distortions—surfaces stretch, bend, or merge abnormally.
  • Structured Misalignment vs. Information Loss: At moderate intervention strengths, semantic corruption doesn't just destroy the object into random noise. Instead, it produces "structured failures" where the geometry deforms drastically but still retains recognizable visual traces of the source object.
  • Contextual Coupling: Even when an intervention is strictly masked to a single object, downstream feature fusion causes the geometric corruption to slightly bleed into neighboring background regions (creating "halos"), proving that object representations are contextually coupled within the network.

Key Results

The primary result is a validated diagnostic tool that successfully maps out continuous "reconstruction-response trajectories" as intervention strength increases.

The framework's practical utility is demonstrated through DecArt, an interactive screen- and VR-based installation. Users can manipulate a slider to alter $\lambda$ in real-time, watching a 3D scene degrade smoothly without jarring coordinate jumps, providing an intuitive look at how deep networks organize spatial data.

Fig. 1: Controlled representation intervention in feed-forward 3D reconstruc- tion. Corrupting only the selected internal features produces a concentrated geometric and appearance change in the target, while the canonical background remains fixed. The interactive interface exposes the intervention strength and supports continuous inspection of the resulting 3D response.

Limitations and Future Work

  • Qualitative Scope: The findings are currently hypothesis-generating and limited to a single feed-forward reconstruction family. Broadening these insights into a universal theory will require multi-model quantitative benchmarks and geometric frequency metrics.
  • ICP Limitations: The background anchoring step corrects for rigid coordinate drift but cannot fix local deformations or missing geometry caused by the interventions.
  • Boundary Leakage: Downsampling the 2D segmentation masks to match the DINOv2 token grid introduces boundary uncertainties, contributing to the observed contextual leakage.

Summary updated

Abstract

Feed-forward 3D reconstruction converts visual observations into explicit spatial representations by combining low-level image processing with high-level features from pretrained visual encoders. When used as a spatial perception module in embodied agents or XR systems, however, final reconstruction quality alone reveals little about how these internal representations constrain predicted geometry or how perturbations to them propagate through the reconstructed scene. We present a controlled intervention framework that perturbs a fixed reconstruction backbone at two representational loci: the shallow convolutional processor and DINOv2 patch features. Processor-level interventions attenuate high-frequency kernel components, inject spatially correlated weight noise, and sparsify parameters; semantic interventions change feature direction, add stochastic perturbations, and can be restricted to tokens associated with a selected object. Varying an intervention strength $\lambda$ yields reconstruction-response trajectories that expose how internal changes propagate into 3D geometry and appearance. Independent backbone executions, however, introduce coordinate-frame drift that confounds cross-strength comparison. We therefore decompose target and background geometry, register sparse intervention anchors to a canonical non-intervened background, and interpolate corresponding target points through their shared pixel indexing. Our evidence is qualitative and hypothesis-generating: existing cases show structured, locus-dependent, and spatially localized failures rather than uniform output noise. We finally demonstrate the framework in DecArt, an interactive screen- and VR-based installation that exposes these spatial response trajectories and interprets representation corruption as computational forgetting. **Keywords:** Spatial perception, Model intervention, Feed-forward 3D reconstruction, Feature corruption, XR
BibTeX
@inproceedings{schlenker2026probing,
  author = {Matti Marino Schlenker and Fran Zezelj and Philipp M. H. Bernhardt and Sophie Anderle and Polina Karpikova and Gege Gao},
  title = {{Probing Spatial Representations in Feed-Forward 3D Reconstruction through Controlled Intervention}},
  booktitle = {ODEWM @ ECCV},
  year = {2026},
  url = {https://openreview.net/forum?id=fr5vqAwl9b},
}
Figure