Ray-Based Camera Conditioning for Spatial Neural Networks

68 concepts55 papers1 proposal

When training deep networks to generate or reason about 3D scenes, the model must understand where the camera is positioned. Traditionally, this is achieved using a global camera matrix—a $3\times4$ or $4\times4$ transformation matrix encoding the camera's rotation and translation. While mathematically complete, this representation introduces a severe impedance mismatch when fed into spatial architectures like Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs).

A global matrix is an abstract, low-dimensional numerical block. When injected into a network via global cross-attention or MLPs, every individual pixel or patch must independently decode this matrix to understand its own local geometric context. The network is forced to implicitly learn the complex, non-linear physics of perspective projection from scratch. As a result, models often struggle with precise camera control, leading to spatial distortions, perspective errors, or "warping" artifacts during camera movements.

Translating Matrices into Spatial Ray Maps

To bridge this gap, a rapidly growing body of research shifts the geometric conditioning signal from the global level to the pixel level. Instead of a single matrix, the camera pose is translated into an image-sized map of 3D rays, where each pixel or patch contains explicit geometric parameters describing the light ray passing through it.

As shown by Carl Doersch et al. in Input-level Inductive Biases for 3D Reconstruction, treating camera geometry as dense, per-pixel tokens rather than raw projection matrices allows general-purpose architectures to learn multi-view relationships far more efficiently.

To represent these rays uniquely and without coordinate ambiguity, researchers frequently leverage Plücker coordinates. A standard ray representation using an origin and a direction vector is redundant because any point along the ray can serve as its origin. Plücker coordinates resolve this by encoding a 3D line as a 6D vector consisting of its direction $\mathbf{d}$ and its moment $\mathbf{m} = \mathbf{o} \times \mathbf{d}$ (where $\mathbf{o}$ is the camera center).

This 6D formulation provides a redundant-free, continuous spatial encoding for every pixel. Eric Ming Chen et al. utilized this in Ray Conditioning: Trading Photo-consistency for Photo-realism in Multi-view Image Generation to build a "geometry-free" multi-view generator, allowing a 2D network to interpret the geometric context of a scene without building an explicit 3D proxy.

Why It Works: Aligning Geometry with Visual Structure

The core reason ray conditioning works so well is structural alignment. By turning abstract camera parameters into a 2D spatial map that matches the dimensions of the image or latent feature space, geometry becomes "visual data."

  • Local Processing of Global Trajectories: CNNs and Transformers excel at processing local spatial correlations. When camera trajectories are represented as pixel-wise ray maps, convolutional layers or local attention mechanisms can interpret them directly as spatial features (Hao He et al., CameraCtrl: Enabling Camera Control for Video Diffusion Models).
  • Direct Mapping to Pixel Shifts: A network can easily learn how a change in a pixel's 3D ray vector correlates with a 2D pixel shift across frames. As demonstrated by Guojun Lei et al. in MotionFlow:Learning Implicit Motion Flow for Complex Camera Trajectory Control in Video Generation, this 6D per-pixel representation provides a direct mapping between 3D geometry and 2D image space, significantly improving motion accuracy and reducing spatial distortion.
  • Resolution of Projection Ambiguities: Traditional methods matching discrete 2D points to 3D space suffer from scale and projection ambiguities. Modeling image patches as continuous 3D ray bundles originating from the camera center naturally mirrors the physical geometry of a pinhole camera (Xinjun Li et al., RayI2P: Learning Rays for Image-to-Point Cloud Registration).
  • Unification of Intrinsics and Extrinsics: Instead of treating field-of-view or focal length as separate numerical parameters, converting intrinsics directly into ray patches allows models to maintain geometric consistency across varying lens configurations and high-resolution image crops, as demonstrated in Pow3R (Philippe Weinzaepfel et al.) and Multi-HMR (Fabien Baradel et al.).

Applications Across Vision and Graphics

This paradigm has unlocked state-of-the-art capabilities across several domains:

Camera-Controllable Video Diffusion

Injecting Plücker ray embeddings into video foundation models allows users to dictate complex cinematic trajectories. For instance, Sherwin Bahmani et al. developed VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control, which aligns pixel-level geometric rays with the spatial structure of visual tokens in video transformers. Similarly, Zhijing Cheng et al. introduced MoCa: Modeling Object Consistency for 3D Camera Control in Video Generation, mapping camera trajectories directly into the 2D latent space to eliminate object warping. This dense parameterization is also crucial for human animation, as shown in HumanVid (Yixuan Li et al.) and CamCo (Dejia Xu et al.).

Novel View Synthesis and Feed-Forward 3D Reconstruction

In novel view synthesis (NVS), ray conditioning allows models to generalize to arbitrary camera trajectories without relying on explicit rendering equations. Haian Jin et al. demonstrated in LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias that Plücker ray embeddings enable a purely data-driven transformer to learn spatial relationships directly from data. For feed-forward 3D reconstruction models like GS-LRM (Kai Zhang et al.), tokenizing camera poses as Plücker rays allows the transformer backbone to seamlessly reason about the geometric relationship between image patches and the 3D space they represent.

Cross-Sensor and Multi-View Alignment

Beyond generation, ray embeddings act as a powerful geometric prior for multi-camera systems. In autonomous driving settings, Florent Bartoccioni et al. introduced LaRa: Latents and Rays for Multi-Camera Bird’s-Eye-View Semantic Segmentation, which encodes the physical origin and direction of light rays to relate features across disjoint camera views within a shared 3D coordinate system. Similarly, Simon Doll et al. utilized 3D view rays in SpatialDETR to provide a geometric inductive bias, allowing cross-sensor attention mechanisms to seamlessly detect objects across camera boundaries.

Key Foundational Work

SpatialDETR: Robust Scalable Transformer-Based 3D Object Detection From Multi-view Camera Images With Global Cross-Sensor Attention (2022) — contributed 2 concept elements to this direction.
Input-level Inductive Biases for 3D Reconstruction (2022) — contributed 2 concept elements to this direction.
LaRa: Latents and Rays for Multi-Camera Bird’s-Eye-View Semantic Segmentation (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_plucker_coordinate_embedding_abee840f,
  title = {Plücker coordinate embedding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/plucker-coordinate-embedding/rdr_abee840f5a2a4d76bd5f071aab7347f7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_plucker_coordinate_embedding_abee840f_v1,
  title = {Plücker coordinate embedding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/plucker-coordinate-embedding/rdr_abee840f5a2a4d76bd5f071aab7347f7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake