TL;DR
Researchers have discovered that diverse vision models—ranging from Stable Diffusion to DINOv2—internally represent 3D viewing direction as a simple 2D plane in their hidden feature space. Crucially, these models can "extrapolate" this understanding to camera angles they weren't specifically probed on, outperforming simple geometric baselines and suggesting they learn a fundamental, albeit local, representation of 3D space.
What Problem Does This Paper Address?
Vision models are trained on 2D images, but those images are projections of a 3D world. A long-standing question in AI is whether these models actually "understand" 3D structure (like camera viewpoint) or if they simply learn 2D "shortcuts"—for example, associating a specific shadow shape with a label without understanding the spatial relationship between the light, the object, and the camera.
The challenge in testing this is that in most datasets, when the camera moves, the objects move to different pixels. This makes it hard to tell if a model is tracking 3D direction or just 2D pixel coordinates. This paper introduces a "fixed-projection" setup to isolate viewing direction from layout, asking: if the objects stay at the exact same pixels but the perspective and shading change, does the model still know where the camera is?
Methods
The authors employ several clever techniques to probe the "frozen" (unmodified) internal states of eight major models (including PixArt-$\alpha$, Stable Diffusion 3, DINOv2, and CLIP):
- Fixed-Projection Dataset: They rendered scenes where two objects are locked to the same two pixel coordinates regardless of camera movement. This forces the model to rely on subtle cues like shading, specular highlights, and shadow deformation to determine the viewing angle ($\alpha$).
- Shared-Plane Probing: Instead of a standard linear readout, they tested a specific geometric hypothesis: that the difference between the features of two objects traces a circle $(\cos \alpha, \sin \alpha)$ on a 2D plane within the high-dimensional hidden space.
- Coverage-Controlled Probing: To distinguish between "memorization" and "representation," they trained probes on one set of camera paths (circles) and tested them on entirely different paths (ellipses). They specifically looked at "outside coverage" areas—viewpoints the probe had never seen during its training.
- Mechanistic Interventions: In the PixArt-$\alpha$ model, they didn't just watch the features; they actively edited them (rotating or ablating the discovered plane) to see if it changed the generated image's viewpoint.
Datasets
- Tier A (Primitives): Rendered spheres, cylinders, and cubes with fixed projections.
- Tier B (Appearance Variants): The same scenes but with randomized lighting, colors, and floor textures to test the robustness of the encoding.
- Tier C (Real Objects): 500 household items from the Google Scanned Objects dataset, path-traced with realistic environment lighting (HDRIs) to see if the "plane" exists in complex, real-world geometry.
Key Insights and Ideas
- The Universal Plane: All eight models, despite having different architectures and training objectives (diffusion vs. self-supervised vs. image-text), represent relative viewing direction in the same strikingly simple way: a 2D plane.
- Extrapolation is the "Acid Test": While a simple geometric baseline (tracking shadow positions) can predict the camera angle on familiar paths, it fails miserably (63° error) on new paths. In contrast, the vision models' internal features remain accurate (6–8° error), proving they've captured a more generalizable concept of viewpoint.
- Abrupt Emergence: In the PixArt-$\alpha$ transformer, this 3D representation doesn't "fade in." It forms abruptly at a specific layer (Block 6) and is then passed downstream.
- Local vs. Global Disconnect: This is perhaps the most surprising finding. While the model has a clear internal representation of viewpoint at the object level, editing that representation doesn't rotate the whole scene. Instead, it changes the object's appearance (e.g., its color or texture). This suggests that the "global" viewpoint is determined by a "vote" across the entire image (including the floor and background), and the local object features are just one part of that consensus.
Key Results
- Accuracy: Seven of the eight models extrapolated to unseen camera positions with high precision. For example, PixArt-$\alpha$ achieved a mean angular error of only 6.5° outside its training coverage, compared to 101.6° for a linear geometric baseline.
- Robustness: The orientation of the "viewing plane" remained consistent even when the researchers changed the objects' colors, the floor texture, or the lighting direction.
- Plane Strength: In Stable Diffusion 3, the shared plane explained up to 87% of the variance in the features that changed with the viewpoint.
- Real-World Transfer: While a single "universal" plane didn't emerge for all real-world objects (Tier C), each specific pair of real objects still formed its own internal 2D plane, showing the geometry holds even for non-primitive shapes.
Limitations and Future Work
- Elevation Confounding: The models struggled to extrapolate when the camera's elevation changed significantly, as elevation cues were often confounded with the horizontal viewing angle in the training data.
- The "Control" Problem: The fact that intervening on the plane didn't rotate the generated scene is a major limitation for those hoping to use these features for "steering" image generation. Future work is needed to find the "global" viewpoint variable that coordinates the objects, shadows, and floor.
- Scaling to Complex Scenes: The study focused on pairs of objects. How this representation scales to crowded, naturalistic scenes with many interacting objects remains an open question.
Reference: Ding, P. (2026). Beyond the Training Cameras: Coverage-Controlled Probes of Viewing Direction in Frozen Vision Models. arXiv:2611.19642v1.
Summary updated