Viewer Centered Representations for Generalizable 3D Vision

46 concepts30 papers1 proposal

In traditional 3D computer vision, models often rely on a "canonical" coordinate system. To estimate the pose of a car, a model is typically trained to recognize a standardized "front," "side," and "top." While effective for familiar categories, this approach fails when the model encounters a novel object—like a uniquely shaped sculpture or a new piece of furniture—where no predefined "front" exists.

A growing body of research suggests that the key to generalizable 3D understanding lies in shifting the perspective from the object to the observer. By using viewer-centered coordinate frames and relative transformations, models can reason about geometry and orientation without needing to know what the object is beforehand.

The Problem with Canonical Shortcuts

The reliance on object-centered coordinates often allows neural networks to take "shortcuts." As Tatarchenko et al. demonstrated, many single-view 3D reconstruction networks do not actually learn to "lift" 2D images into 3D geometry. Instead, they perform high-level classification, recognizing a "chair" and retrieving a similar shape from memory. Because all chairs in the training set are aligned to the same canonical frame, the model simply outputs a memorized template.

To force a model to learn genuine geometric reasoning, researchers have proposed performing reconstruction in a viewer-centered coordinate system. In this setup, the origin is defined by the camera's perspective rather than the object's center. Shin showed that viewer-centered frames significantly improve generalization to unseen categories because the model must learn to infer geometry from visible surfaces and camera rays rather than relying on category-specific template retrieval.

Relative Pose as a Geometric Anchor

If we cannot rely on a fixed "front" for a novel object, how do we define its orientation? The solution is to treat pose as a relative relationship between two views.

Instead of predicting an absolute orientation, Vaze et al. framed pose estimation as a relative alignment task between two instances of a category. This bypasses the need for standardized coordinate systems entirely. Similarly, Kuznetsova et al. demonstrated that by calculating the relative orientation distance to nearest neighbors in a metric space, a model can achieve "zero-shot" pose prediction for objects it has never seen in training.

This shift enables models to handle "in-the-wild" scenarios where camera poses are arbitrary. For example, Zhang et al. introduced a framework that predicts probabilistic relative rotations between sparse image pairs, allowing for robust 3D reconstruction of arbitrary objects without requiring a global coordinate system.

Learning from Unaligned Data

A major bottleneck in 3D vision is the need for labor-intensive, human-labeled datasets where every object is perfectly aligned. Viewer-centric approaches alleviate this by enabling origin-free learning.

Guestrin et al. proposed a framework that utilizes the relative transformation between two camera viewpoints as a supervisory signal. By enforcing equivariance—ensuring the latent representation transforms exactly like the physical scene—the model learns to render novel views without ever seeing an absolute global coordinate. This principle is further extended in generative modeling; Gao showed that training diffusion models in a viewer-centric "view space" allows for high-quality 3D-aware image synthesis on unaligned, real-world datasets where a shared canonical frame is impossible to define.

Generalizing Through Local Perspective

The intuition behind why viewer-centric models generalize so well is that the local appearance of a surface relative to a camera ray is often consistent across different objects. De Wolf et al. exploited this by estimating "allocentric" poses—the local viewing angle relative to the camera ray. This ensures that the predicted orientation is independent of where the object appears in the image frame, making the model robust to different camera lenses and fields of view.

In the context of Neural Radiance Fields (NeRF), Yu et al. (pixelNeRF) and Lin et al. (VisionNeRF) demonstrated that performing all 3D operations within the camera coordinate system allows these models to generalize across diverse scenes. By conditioning the radiance field on local image features relative to the observer, these systems can synthesize novel views from a single unposed image, effectively "hallucinating" the far side of an object based on geometric primitives rather than category labels.

From Templates to Relationships

Ultimately, this research direction moves computer vision away from a library of 3D templates toward a flexible understanding of spatial relationships. Whether it is predicting how object relationships change as a viewer moves (Xu et al.) or reconstructing hidden back-sides by predicting cross-sectional thickness relative to the sensor (Nicastro et al.), the common thread is the rejection of the fixed global frame. By anchoring geometry to the viewer, these models achieve a level of flexibility that brings us closer to how humans perceive and interact with an unfamiliar 3D world.

Key Foundational Work

Exploiting View-Specific Appearance Similarities Across Classes for Zero-Shot Pose Prediction: A Metric Learning Approach (2016) — contributed 2 concept elements to this direction.
View-independent scene acquisition for tele-presence (2000) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_relative_pose_estimation_97711aa5,
  title = {relative pose estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-pose-estimation/rdr_97711aa54f684d75919ea0cb3ef9ff3e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_relative_pose_estimation_97711aa5_v1,
  title = {relative pose estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-pose-estimation/rdr_97711aa54f684d75919ea0cb3ef9ff3e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake