TALO: Pushing 3D Vision Foundation Models Towards Globally Consistent Online Reconstruction

Fengyi Zhang, Tianjun Zhang, Kasra Khosoussi, Zheng Zhang, Zi Huang, Yadan Luo

Introduction

3D Vision Foundation Models (3DVFMs) have emerged as powerful tools for dense 3D geometry and camera pose prediction from uncalibrated images. Models like VGGT, $\pi^3$, and MapAnything demonstrate impressive zero-shot capabilities in single-pass reconstructions. However, their application in real-world online scenarios—such as autonomous driving—remains challenging. In these settings, data is processed in temporal windows or "submaps," and maintaining global consistency across these independently inferred segments is a major hurdle. TALO (Thin-plate Alignment for Long-term Online reconstruction) addresses this by introducing a flexible, non-linear alignment framework.

The Challenge of Online Reconstruction

Online reconstruction requires stitching consecutive submaps into a unified 3D scene. Traditional approaches rely on global linear transformations, such as 7-DOF Sim(3) or 15-DOF SL(4) warps. These methods assume that geometric errors are uniform across the scene, which is rarely the case for 3DVFMs. Distortions often vary spatially, such as different depth scales across different cameras in a surround-view rig.

Figure 1: Limitations of linear alignment methods like Sim(3) and SL(4), which often lead to divergent geometry and implausible poses in complex outdoor scenes.

As shown in Figure 1, forcing a single global linear warp often results in "tearing" the geometry or overfitting to specific areas while neglecting others. Furthermore, these methods typically align only adjacent submaps, leading to significant long-term drift. The reliance on noisy point clouds for alignment often causes mathematical instability, particularly for high-DOF warps like SL(4), which frequently diverge in outdoor environments.

The TALO Framework

TALO replaces rigid, short-term alignment with a robust, long-term deformation framework based on Thin Plate Splines (TPS). The approach is built on three technical pillars:

1. Point-Agnostic Submap Registration

To avoid the pitfalls of aligning noisy 3D point clouds, TALO utilizes the relative transformations of overlapping camera poses. Since 3DVFMs provide relatively stable pose estimates, averaging these transformations provides a highly stable initial registration that is immune to the geometric noise that derails ICP-style methods.

2. Global Control Point Propagation

TALO extracts sparse 3D control points through voxelization. These points are tracked forward and backward across multiple submaps using the foundation model's pixel-aligned predictions. This builds a rich, globally connected graph of observations for the exact same physical locations over time.

Figure 4: The TALO methodology, illustrating (c) global control point propagation across submaps and (d) the resulting Thin Plate Spline deformation field used for global consistency.

3. Thin Plate Spline (TPS) Deformation

By aggregating multi-submap observations, TALO determines a "canonical" 3D position for each control point. A 3D TPS deformation field is then fitted to warp the submap points into a globally consistent space. TPS provides the necessary spatial adaptability to correct local distortions while mathematically preserving local rigidity so the scene structure remains intact.

Results and Impact

TALO was evaluated on the Waymo Open and nuScenes datasets, demonstrating superior performance across multiple foundation models including VGGT, $\pi^3$, and MapAnything.

  • Unmatched Robustness: TALO achieved zero catastrophic failures across all datasets. In contrast, SL(4)-based methods failed in over 60% of tested outdoor scenes due to geometric divergence.
  • Trajectory Accuracy: The system consistently reached the lowest Absolute Trajectory Error (ATE), averaging ~1 meter on both major datasets.
  • Massive Rotational Improvements: On the Waymo dataset, TALO reduced Relative Rotation Error (RRE) from 0.71° to just 0.14°—a 5x improvement that practically eliminates long-term trajectory drift.
  • Cleaner Geometry: Qualitatively, TALO eliminates "multi-layer artifacts" like ghosted facades and duplicated vehicles that occur when prior methods fail to perfectly align overlapping submaps.

Conclusion

TALO demonstrates that non-linear, spatially adaptive warping is essential for online 3D reconstruction using foundation models. By prioritizing pose-based registration and global control point consistency, TALO provides a plug-and-play solution that significantly enhances the reliability and accuracy of 3DVFMs in dynamic, real-world environments.

Summary updated

BibTeX
@inproceedings{zhang2026talo,
  author = {Fengyi Zhang and Tianjun Zhang and Kasra Khosoussi and Zheng Zhang and Zi Huang and Yadan Luo},
  title = {{TALO: Pushing 3D Vision Foundation Models Towards Globally Consistent Online Reconstruction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=6LH4I0fFOf},
}
Figure