Rig3R: Rig-Aware Conditioning and Discovery for 3D Reconstruction

Samuel Li, Pujith Kachana, Prajwal Chidananda, Saurabh Nair, Yasutaka Furukawa, Matthew BrownNeurIPS 2025 · OpenReview

TL;DR

Rig3R is a transformer-based 3D reconstruction model that leverages "rig" structure—the fixed spatial relationship between cameras on a vehicle or robot. By conditioning on optional metadata and introducing a novel "raymap" output, it achieves state-of-the-art accuracy in pose estimation and 3D mapping. Remarkably, it can even discover a rig's geometry from scratch when no metadata is provided.

The Challenge of Multi-Camera Rigs

In autonomous driving and robotics, agents rarely rely on a single camera. Instead, they use "rigs"—synchronized clusters of cameras bolted to a rigid frame. Classical computer vision techniques, like Structure-from-Motion (SfM), excel at utilizing these fixed constraints. However, modern learned models (like DUSt3R) often treat input images as an unstructured "bag of photos."

This approach represents a significant missed opportunity. If a model doesn't inherently know that Camera A is always exactly 50cm to the left of Camera B, it has to work much harder to align their views. This lack of structural awareness makes current models brittle in scenarios with minimal view overlap or in feature-poor regions like the open sky, where traditional matching fails.

Rig3R takes multi-camera images and optional metadata to produce state-of-the-art 3D reconstructions and pose estimates.

How Rig3R Works

Rig3R generalizes prior multiview models by making them explicitly rig-aware. Its architecture consists of a shared ViT-Large encoder and a transformer decoder that performs joint self-attention across all input frames.

The model's success relies on two core innovations:

  1. Metadata Conditioning with Dropout: Rig3R can accept optional metadata—such as Camera IDs, timestamps, and rig poses—as input embeddings. Crucially, the authors employ "embedding dropout" during training. This technique teaches the model to utilize metadata when available, but also forces it to learn how to infer that structural relationship directly from the images when metadata is missing.
  2. Raymaps over Pointmaps: Instead of solely predicting 3D pointmaps, Rig3R predicts raymaps—dense directional fields where every pixel is assigned a unit ray originating from a shared camera center.
  3. Pose Raymaps represent the camera's position in a global frame, effectively handling ego-motion.
  4. Rig Raymaps represent the camera's position relative to the rig itself, capturing the fixed geometry.

These raymaps allow for the "closed-form" recovery of camera intrinsics and extrinsics. This means the math to find the camera's position is direct and does not require iterative guessing or complex optimization loops.

Rig3R architecture showing metadata conditioning and the generation of pointmaps, pose raymaps, and rig raymaps.

Datasets and Evaluation

The model was trained on a massive, diverse mix of datasets including CO3D-v2, BlendedMVS, ScanNet++, nuScenes, and Waymo.

For evaluation, the authors focused on two rigorous real-world benchmarks:

  • Waymo Open Dataset: Utilizes LiDAR-based ground truth for highly accurate poses and 3D points.
  • WayveScenes101: A challenging dataset featuring a 5-camera rig and approximately 200 timesteps per scene, specifically used to test zero-shot generalization to unseen rig configurations.

Key Insights

  • The "Jagged Rig" Advantage: Most multi-camera models expect a perfect grid of data (all cameras firing at all times). Rig3R uses a "jagged" sampling strategy, flattening all frames into a single sequence. This allows it to naturally handle missing frames, irregular timestamps, or asynchronous captures.
  • Raymaps > Pointmaps for Pose: The authors found that raymaps are far more stable for estimating camera positions than 3D pointmaps. Pointmaps struggle with the sky or reflective surfaces where depth is undefined. Raymaps, which only encode direction, provide a consistent, reliable signal across the entire image.
  • Rig Discovery: Perhaps the most impressive feat is Rig3R's "Rig Discovery" capability. Even when fed a pile of unordered images with zero metadata, the model can cluster images from the same physical camera and reconstruct the rig's geometry entirely from visual context.

Key Results

Rig3R sets a new standard for feed-forward 3D vision:

  • Pose Estimation: On the Waymo dataset, Rig3R achieved an 82.1% mAA (mean Average Accuracy), significantly outperforming baselines like Fast3R (20.6%) and DUSt3R-GA (37.5%).
  • Reconstruction Quality: It reduced the Chamfer distance (a measure of 3D error) by orders of magnitude compared to previous learned methods (0.2 on Waymo vs. 3.9 for Fast3R).
  • Zero-Shot Generalization: On the unseen WayveScenes101 dataset, it outperformed traditional COLMAP pipelines, proving that its learned rig-priors generalize well to entirely new vehicle setups.
  • Speed: All of this is achieved in a single forward pass, eliminating the need for expensive post-processing or iterative bundle adjustment.

Limitations and Future Work

While Rig3R represents a significant leap forward, the authors note a few areas for future development:

  • Data Diversity: The model's performance remains tied to the variety of rigs it observes during training. Future work could leverage data augmentation to simulate an even wider array of "virtual rigs."
  • Dynamic Objects: While raymaps are robust, the model does not explicitly model independent motion (e.g., other cars moving in the scene). Integrating temporal motion models could further sharpen reconstructions in heavy traffic.
  • Scaling: While Rig3R handles 24-frame chunks efficiently, scaling to hundreds of frames with perfect global consistency remains a computational challenge for transformer architectures.

Summary updated

Abstract

Estimating agent pose and 3D scene structure from multi-camera rigs is a central task in embodied AI applications such as autonomous driving. Recent learned approaches such as DUSt3R have shown impressive performance in multiview settings. However, these models treat images as unstructured collections, limiting effectiveness in scenarios where frames are captured from synchronized rigs with known or inferable structure. To this end, we introduce Rig3R, a generalization of prior multiview reconstruction models that incorporates rig structure when available, and learns to infer it when not. Rig3R conditions on optional rig metadata including camera ID, time, and rig poses to develop a rig-aware latent space that remains robust to missing information. It jointly predicts pointmaps and two types of raymaps: a pose raymap relative to a global frame, and a rig raymap relative to a rig-centric frame consistent across time. Rig raymaps allow the model to infer rig structure directly from input images when metadata is missing. The global pose raymaps allow the model to reason about the agent’s ego-motion, while the rig raymaps allow the model to infer rig structure directly from input images when metadata is missing. Rig3R achieves state-of-the-art performance in 3D reconstruction, camera pose estimation, and rig discovery -- outperforming both traditional and learned methods by 17-45% mAA across diverse real-world rig datasets, all in a single forward pass without post-processing or iterative refinement.
BibTeX
@inproceedings{li2025rig,
  author = {Samuel Li and Pujith Kachana and Prajwal Chidananda and Saurabh Nair and Yasutaka Furukawa and Matthew Brown},
  title = {{Rig3R: Rig-Aware Conditioning and Discovery for 3D Reconstruction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=vEFPm6gw2s},
}
Figure