Towards Live 3D Reconstruction from Wearable Video: An Evaluation of V-SLAM, NeRF, and Videogrammetry Techniques

David Ramirez, Suren Jayasuriya, Andreas SpaniasarXiv 2022 · DOI · arXiv

Abstract

The creation of real-time "digital twins" from wearable sensors is a critical requirement for immersive Mixed Reality (MR) military training. This study evaluates three primary 3D reconstruction paradigms: Structure-from-Motion (SfM) via COLMAP, Visual-SLAM (V-SLAM) via ORB-SLAM3, and Neural Radiance Fields (NeRF) via Instant-NGP. By benchmarking these techniques on a mid-range GPU, the research identifies a significant performance-density tradeoff. While ORB-SLAM3 achieves true real-time performance (24ms per frame), its sparse output is insufficient for detailed physical simulations. Conversely, NeRFs offer high visual density but face challenges with sequential, frame-by-frame processing. The paper proposes a hybrid pipeline leveraging V-SLAM for positioning and NeRF for dense environmental mapping.

Introduction: The Digital Twin Challenge

To enable effective Mixed Reality (MR) training, commanders require a high-fidelity 3D model of the physical environment that updates dynamically based on soldier-worn sensors. This "digital twin" allows for the integration of simulated agents and the calculation of digital ballistics within a real-world context.

Conceptual overview of a Digital Twin environment for military training, where physical actions are mirrored in a simulated space.

The core challenge is the computational tradeoff between speed and detail. Traditional photogrammetry produces accurate models but requires hours of offline processing. Visual-SLAM is fast enough for navigation but generates "sparse" point clouds—skeletal structures that lack the surface detail needed to simulate physical interactions, such as a digital bullet hitting a specific wall.

Methodology and Benchmarking

The researchers evaluated three distinct algorithmic approaches on an RTX 2060 GPU:

  1. COLMAP (SfM): An offline, CPU-intensive method using SIFT features for high-accuracy triangulation.
  2. ORB-SLAM3 (V-SLAM): A robotics-focused approach prioritizing speed by tracking minimal feature points on a single CPU core.
  3. Instant-NGP (NeRF): A neural rendering technique that trains a small network to represent scene volume, capable of rapid learning but typically requiring pre-calculated camera poses.
Multi-view reconstruction concept: integrating images from various angles to build a 3D representation.

Evaluation was primarily conducted using the KITTI Vision Benchmark Suite, which provides dashboard video synchronized with Lidar ground truth. The authors also introduced "stress test" datasets involving high-speed paintball simulations and body-worn cameras in natural environments.

Results: The Density vs. Speed Wall

Quantitative analysis revealed a massive performance gap between the methods:

Algorithm Total Time per Frame Output Characteristics
ORB-SLAM3 0.024 seconds Real-time; Sparse point cloud
Instant-NGP (NeRF) 0.590 seconds Near real-time; Dense volume
COLMAP 1.33 seconds Offline; High-fidelity mesh

ORB-SLAM3 processed frames in just 24ms, making it the only "true" real-time candidate, though its maps were too thin for complex MR interactions. Instant-NGP showed promise with substantial visual improvements over traditional geometric methods, though it currently struggles with the "sequential hurdle"—the need to process video frames incrementally rather than in batches.

Discussion and Future Directions

A key insight from the study is the potential for symbiotic pipelines. By using ORB-SLAM3 to provide rapid camera positioning data, a NeRF model could be fed "live" to paint a dense 3D environment incrementally.

However, significant hurdles remain, particularly regarding motion blur from erratic head movements and the lack of absolute real-world scale in monocular video. Future research will focus on testing these pipelines in the chaotic, high-stakes environment of their paintball dataset to ensure the algorithms can survive the rapid movement of a soldier in a firefight.

Conclusion

While no single technology currently meets all requirements for a live military digital twin, the combination of V-SLAM's speed and NeRF's density offers a viable path forward. Bridging the gap between these paradigms will be essential for the next generation of immersive training environments.

Summary updated

Abstract

Mixed reality (MR) is a key technology which promises to change the future of warfare. An MR hybrid of physical outdoor environments and virtual military training will enable engagements with long distance enemies, both real and simulated. To enable this technology, a large-scale 3D model of a physical environment must be maintained based on live sensor observations. 3D reconstruction algorithms should utilize the low cost and pervasiveness of video camera sensors, from both overhead and soldier-level perspectives. Mapping speed and 3D quality can be balanced to enable live MR training in dynamic environments. Given these requirements, we survey several 3D reconstruction algorithms for large-scale mapping for military applications given only live video. We measure 3D reconstruction performance from common structure from motion, visual-SLAM, and photogrammetry techniques. This includes the open source algorithms COLMAP, ORB-SLAM3, and NeRF using Instant-NGP. We utilize the autonomous driving academic benchmark KITTI, which includes both dashboard camera video and lidar produced 3D ground truth. With the KITTI data, our primary contribution is a quantitative evaluation of 3D reconstruction computational speed when considering live video.
BibTeX
@misc{ramirez2022towards,
  author = {David Ramirez and Suren Jayasuriya and Andreas Spanias},
  title = {{Towards Live 3D Reconstruction from Wearable Video: An Evaluation of V-SLAM, NeRF, and Videogrammetry Techniques}},
  year = {2022},
  eprint = {2211.11836},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.11836},
}
Figure