TL;DR
RECALL is a feed-forward neural network that recalibrates multi-camera rigs in real-time (~0.12 ms) by predicting small drifts from factory settings and solving for global rig rotation. It matches the accuracy of offline bundle adjustment while being five orders of magnitude faster, requiring only a brief observation of a known object.
What Problem Does This Paper Address?
High-precision multi-camera systems—found in everything from AR/VR headsets to autonomous vehicles and motion-capture studios—ship with factory (OEM) calibrations. However, these calibrations are fragile. Thermal expansion, mechanical vibration, and simple handling cause the cameras to "drift" from their nominal values. Furthermore, every time a rig is moved or remounted, its global orientation relative to the world changes.
Fixing this traditionally requires Bundle Adjustment (BA), a global optimization process that is computationally expensive and far too slow for real-time use. While recent "foundation models" for geometry can estimate poses from images, they are often non-metric (they don't know the true scale) and ignore the valuable "anchor" of the original factory calibration. RECALL bridges this gap by providing a metric, real-time solution that exploits the rig's known structure.
Methods
The authors propose a "feed-forward" approach, meaning the network predicts the calibration in a single pass rather than through an iterative optimization at runtime.
Key components include:
- OEM-Anchored Residuals: Instead of predicting camera parameters from scratch, the model predicts the drift relative to the factory calibration. This "anchoring" makes the learning task much more stable and well-posed.
- Rig-Coupled Recurrent Refinement: The model doesn't just guess once. It uses a recurrent operator that reprojects the current estimate, calculates residuals, and uses a transformer to apply attention across all cameras. This allows the network to "understand" the rigidity of the rig.
- Learned Gauss-Newton (GN) Gauge Solver: This is perhaps the most novel part. Global rotation is often "ill-conditioned" for standard neural network regression. RECALL embeds a differentiable 3x3 Gauss-Newton solver inside the network to analytically solve for the shared global rotation at each refinement step.
- On-the-Fly Synthesis: Because camera geometry follows known mathematical laws, the authors don't need a massive dataset of real images. They train the model entirely on synthetic data generated by perturbing the OEM calibration and projecting a known object.
Datasets
The model was evaluated on both synthetic and real-world benchmarks:
- Synthetic 10-Camera Benchmark: An O-shaped rig observing a cube, used to establish baseline performance against traditional BA.
- TotalCapture: A real-world dataset featuring an 8-camera rig with Vicon ground truth.
- CMU Panoptic: A massive 10-camera HD surround rig. This was used to test "synthetic-to-real" transfer, where the model trained on synthetic data was evaluated on real-world captures.
Key Insights and Ideas
- Optimization as a Layer: By embedding a Gauss-Newton step in the network, the authors combine the strengths of classical geometry (which handles global rotation well) with deep learning (which handles complex, non-linear drifts well).
- Rigidity as a Prior: Most learned calibration methods treat cameras independently. RECALL uses cross-camera attention to enforce the idea that if one camera moves, the others likely moved in a correlated way because they are bolted to the same frame.
- The Power of Synthesis: The paper demonstrates that for geometric tasks, you don't necessarily need real-world training data. If you can accurately model the physics of the "drift," synthetic data is sufficient for high-accuracy real-world performance.
Key Results
RECALL's performance is a massive leap over traditional methods in terms of efficiency:
- Speed: It runs in approximately 0.12 ms per rig. For comparison, rig-coupled joint BA takes tens of seconds. This represents a $10^5\times$ speedup.
- Accuracy: On a synthetic 10-camera rig, RECALL achieved an absolute rotation error of $0.26^\circ$, outperforming joint BA ($0.47^\circ$) and far surpassing per-camera BA ($2.30^\circ$).
- Real-World Robustness: On the CMU Panoptic dataset, it matched or beat offline BA on rotation metrics while being orders of magnitude faster. Compared to image-only foundation models like VGGT, RECALL was roughly $15\times$ more accurate in estimating focal length because it utilized the OEM anchor.
Limitations and Future Work
While impressive, the authors note several areas for improvement:
- Configuration Specificity: Currently, a model must be trained for a specific rig/object configuration. Future work could explore more generalized models that work across different rig topologies without retraining.
- Bounded Rotation: The model assumes the rig's global reorientation is "bounded" (e.g., tilted within a certain range). It might struggle with a rig that has been completely flipped or spun 360 degrees.
- Outlier Sensitivity: Like most geometric solvers, it is sensitive to "gross outliers"—bad 2D-3D correspondences. Integrating a more robust "front-end" for feature detection would make the system more resilient in messy real-world environments.
Summary updated