RECALL: Real-Time Recalibration of Multi-Camera Rigs via Learned OEM-Anchored Residuals

Mario Alberto Tapia de la Cruz, Charalambos PoullisarXiv 2026 · arXiv

TL;DR

RECALL is a feed-forward neural network that recalibrates multi-camera rigs in real-time (~0.12 ms) by predicting small drifts from factory settings and solving for global rig rotation. It matches the accuracy of offline bundle adjustment while being five orders of magnitude faster, requiring only a brief observation of a known object.

What Problem Does This Paper Address?

High-precision multi-camera systems—found in everything from AR/VR headsets to autonomous vehicles and motion-capture studios—ship with factory (OEM) calibrations. However, these calibrations are fragile. Thermal expansion, mechanical vibration, and simple handling cause the cameras to "drift" from their nominal values. Furthermore, every time a rig is moved or remounted, its global orientation relative to the world changes.

Fixing this traditionally requires Bundle Adjustment (BA), a global optimization process that is computationally expensive and far too slow for real-time use. While recent "foundation models" for geometry can estimate poses from images, they are often non-metric (they don't know the true scale) and ignore the valuable "anchor" of the original factory calibration. RECALL bridges this gap by providing a metric, real-time solution that exploits the rig's known structure.

Methods

The authors propose a "feed-forward" approach, meaning the network predicts the calibration in a single pass rather than through an iterative optimization at runtime.

Key components include:

  • OEM-Anchored Residuals: Instead of predicting camera parameters from scratch, the model predicts the drift relative to the factory calibration. This "anchoring" makes the learning task much more stable and well-posed.
  • Rig-Coupled Recurrent Refinement: The model doesn't just guess once. It uses a recurrent operator that reprojects the current estimate, calculates residuals, and uses a transformer to apply attention across all cameras. This allows the network to "understand" the rigidity of the rig.
  • Learned Gauss-Newton (GN) Gauge Solver: This is perhaps the most novel part. Global rotation is often "ill-conditioned" for standard neural network regression. RECALL embeds a differentiable 3x3 Gauss-Newton solver inside the network to analytically solve for the shared global rotation at each refinement step.
  • On-the-Fly Synthesis: Because camera geometry follows known mathematical laws, the authors don't need a massive dataset of real images. They train the model entirely on synthetic data generated by perturbing the OEM calibration and projecting a known object.
The RECALL operator uses a context encoder and a recurrent refinement loop that includes a learned Gauss-Newton solver for global rotation.

Datasets

The model was evaluated on both synthetic and real-world benchmarks:

  • Synthetic 10-Camera Benchmark: An O-shaped rig observing a cube, used to establish baseline performance against traditional BA.
  • TotalCapture: A real-world dataset featuring an 8-camera rig with Vicon ground truth.
  • CMU Panoptic: A massive 10-camera HD surround rig. This was used to test "synthetic-to-real" transfer, where the model trained on synthetic data was evaluated on real-world captures.

Key Insights and Ideas

  • Optimization as a Layer: By embedding a Gauss-Newton step in the network, the authors combine the strengths of classical geometry (which handles global rotation well) with deep learning (which handles complex, non-linear drifts well).
  • Rigidity as a Prior: Most learned calibration methods treat cameras independently. RECALL uses cross-camera attention to enforce the idea that if one camera moves, the others likely moved in a correlated way because they are bolted to the same frame.
  • The Power of Synthesis: The paper demonstrates that for geometric tasks, you don't necessarily need real-world training data. If you can accurately model the physics of the "drift," synthetic data is sufficient for high-accuracy real-world performance.
Figure 1. Method overview. (a) On-the-fly synthesis: from the OEM calibration we sample a bounded global rotation RG (global reorientation ≤θmax) and small OEM-anchored per-camera drift (≤κ), compose ground-truth cameras via Eq. (2), and project the known object to noisy 2D observations, regenerated every iteration with no captured dataset. (b) At test time the observations and OEM anchor pass once through RECALL, the rig-coupled recurrent operator (Fig. 2), which predicts the full per-camera calibration; during training the differentiable projection supplies the reprojection term, reaching every parameter head.

Key Results

RECALL's performance is a massive leap over traditional methods in terms of efficiency:

  • Speed: It runs in approximately 0.12 ms per rig. For comparison, rig-coupled joint BA takes tens of seconds. This represents a $10^5\times$ speedup.
  • Accuracy: On a synthetic 10-camera rig, RECALL achieved an absolute rotation error of $0.26^\circ$, outperforming joint BA ($0.47^\circ$) and far surpassing per-camera BA ($2.30^\circ$).
  • Real-World Robustness: On the CMU Panoptic dataset, it matched or beat offline BA on rotation metrics while being orders of magnitude faster. Compared to image-only foundation models like VGGT, RECALL was roughly $15\times$ more accurate in estimating focal length because it utilized the OEM anchor.

Limitations and Future Work

While impressive, the authors note several areas for improvement:

  • Configuration Specificity: Currently, a model must be trained for a specific rig/object configuration. Future work could explore more generalized models that work across different rig topologies without retraining.
  • Bounded Rotation: The model assumes the rig's global reorientation is "bounded" (e.g., tilted within a certain range). It might struggle with a rig that has been completely flipped or spun 360 degrees.
  • Outlier Sensitivity: Like most geometric solvers, it is sensitive to "gross outliers"—bad 2D-3D correspondences. Integrating a more robust "front-end" for feature detection would make the system more resilient in messy real-world environments.

Summary updated

Abstract

Multi-camera rigs ship pre-calibrated from the manufacturer, but their calibration goes stale each time the rig is repositioned, and drifts continuously through wear, thermal cycling, and vibration. Recovering it on-the-fly is hard: classical bundle adjustment needs a fresh multi-view capture and a global optimization whose cost grows with the rig, ruling out real-time use. In this paper we present RECALL, a feed-forward method that recalibrates a movable multi-camera rig in a single pass. From the nominal OEM calibration as an anchor and a brief observation of a known object, it predicts each camera's current intrinsics and extrinsics, written as a shared global rotation of the rig composed with a small per-camera drift. Instead of regressing the calibration in one shot, RECALL refines its estimate with a learned, rig-coupled recurrent operator that reprojects through a differentiable projection and applies attention across the cameras at each step, and solves for the shared rotational gauge in-network with a learned Gauss-Newton step rather than regressing it. Because the ground truth is synthesizable, we train entirely on data generated on-the-fly and need no captured dataset. On a synthetic 10-camera benchmark, RECALL is more accurate than per-camera and rig-coupled joint bundle adjustment ($0.26^\circ$ vs $0.47^\circ$ rotation, $5.4$ vs $8.9$ px focal) and runs about $10^5\times$ faster at about $0.12$ ms per rig. On real data (TotalCapture, CMU Panoptic) it matches or beats offline rig-coupled bundle adjustment on absolute and relative rotation while running four-to-five orders of magnitude faster, and on the Panoptic surround rig it also matches the strongest image-only foundation models (VGGT, VGGT-$Ω$, $π^3$) on relative pose while staying metric and about $15\times$ more accurate in focal length. Code: https://github.com/theICTlab/RECALL-multicam
BibTeX
@misc{cruz2026recall,
  author = {Mario Alberto Tapia de la Cruz and Charalambos Poullis},
  title = {{RECALL: Real-Time Recalibration of Multi-Camera Rigs via Learned OEM-Anchored Residuals}},
  year = {2026},
  eprint = {2611.14763},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14763},
}
Figure