Does it work outside this benchmark? Introducing the rigid depth constructor tool

Clément Pinard, Antoine Manzanera

Abstract

Monocular depth estimation models are frequently limited by the specific perspectives of the datasets on which they are trained, such as autonomous driving (KITTI) or indoor environments (NYUv2). This paper presents the Rigid Depth Constructor (RDC), an open-source framework designed to democratize the creation of high-quality depth validation datasets. By decoupling Lidar-based environment scanning from camera-based video acquisition, RDC enables the benchmarking of algorithms in diverse, real-world scenarios—including Unmanned Aerial Vehicle (UAV) flight—without the need for expensive, heavy, and perfectly calibrated rigid rigs.

Introduction: The Benchmark Bottleneck

In computer vision research, models are often "benchmarked into a corner." While datasets like KITTI and NYUv2 are foundational, they represent narrow environmental slices. A model optimized for highway navigation may fail when applied to a drone navigating a forest or a robot in a warehouse.

The primary barrier to expanding these benchmarks is the difficulty of generating "ground truth" depth data. Traditionally, this requires a "rigid rig"—a heavy setup where high-end Lidar and cameras are physically coupled and calibrated. Such hardware is often inaccessible to small teams and impractical for lightweight platforms like consumer drones.

Comparison of depth datasets

As shown in the table above, existing benchmarks often lack the camera orientation and position variation necessary for robust UAV applications. RDC addresses this by removing the requirement for simultaneous, rigid hardware integration.

Methodology: The RDC Pipeline

RDC utilizes a "scan-then-localize" workflow that provides maximum flexibility for researchers. The process begins with an environment scan using any Lidar sensor to create a 3D point cloud. Subsequently, standard video is recorded in the same environment.

The RDC processing pipeline

The RDC pipeline processes this data through several key stages:

  1. Photogrammetry: COLMAP (Structure from Motion) is used on a subset of video frames to build a sparse 3D model.
  2. Registration: This sparse model is aligned to the Lidar "ground truth" using Iterative Closest Point (ICP) algorithms, localizing the camera within the scan.
  3. Linear-Time Localization: To maintain computational efficiency, RDC localizes the remaining video frames against the established model in linear time.
  4. Filtering and Projection: A Savitzky-Golay filter smooths the camera trajectory, and Lidar points are projected onto camera frames to create dense depth maps.

New Benchmarks: Manoir and University Hall

To demonstrate the tool's utility, the authors introduced two new datasets:

  • Manoir Dataset: High-resolution (4K/30fps and 720p/120fps) UAV footage of a mansion, featuring complex geometries like sloped roofs and dense foliage.
  • University Hall Dataset: An indoor scene of a large hall with high ceilings, captured via handheld Lidar and drone-mounted cameras.

These datasets highlight the "context is king" principle. When testing established models like DepthNet and SFMLearner, performance varied significantly between car-level and UAV-level perspectives. For instance, in the Manoir dataset, DepthNet outperformed SFMLearner (MAE of 11.99m vs 18.40m), though both struggled with very low depth values.

Discussion and Limitations

A key insight from this work is the power of hardware decoupling. By mapping an area once with high-end Lidar, researchers can subsequently use any lightweight camera—including smartphones—to generate valid data. Furthermore, the authors observed that vision-based localization (via COLMAP) can sometimes prove less noisy than traditional Lidar/IMU odometry.

However, RDC is currently limited to static scenes; dynamic elements like moving vehicles or swaying trees can corrupt depth maps. Additionally, while the hardware requirements are low, the computational cost of running SfM on thousands of frames remains high, and the ICP registration step still requires human verification.

Conclusion

The Rigid Depth Constructor offers a path forward for researchers looking to validate depth estimation models beyond standard benchmarks. By lowering the hardware barrier to ground truth generation, RDC facilitates the development of more robust, context-aware computer vision systems for diverse robotics and UAV applications.

Summary updated

Abstract

A new framework called Rigid Depth Constructor (RDC) is proposed, allowing a user to create his own dataset for the validation of depth map estimation algorithms in the context of autonomous navigation. Compared to the existing tools that rely on high quality fixed Lidar sensor, RDC is usable in low-cost setups requiring only a camera and any (e.g. handheld, or UAV-carried) Lidar sensor, which implies more flexible - and much faster - scene scan. Furthermore, unlike photogrammetry tools that use sparse RGB views, it can be applied to smooth videos while remaining computationally tractable. The framework includes a test suite to get insightful information from the evaluated algorithm. As examples, validation videos made from UAV footage are provided to evaluate two depth prediction algorithms initially tested on in-car driving video datasets, which shows that the drone context is dramatically different. This supports the need to benchmark depth estimation algorithms on a dataset that fits one’s particular context, which often means creating a brand new one. An open source implementation accompanies the paper, designed to be as user-friendly as possible, to make depth dataset creation possible even for small teams. The key contributions are the following: (1) a complete, open-source and almost fully automatic software application for creating validation datasets with densely annotated depth, adaptable to a wide variety of image, video and range data; (2) selection tools to adapt the dataset to specific validation needs, and conversion tools to other dataset formats; (3) as use case examples, two new real datasets, outdoor and indoor, readily usable in UAV navigation context are provided, and used as test sets in the evaluation of two depth prediction algorithms, using a collection of comprehensive (e.g. distribution based) metrics.
BibTeX
@article{pinard2023does,
  author = {Clément Pinard and Antoine Manzanera},
  title = {{Does it work outside this benchmark? Introducing the rigid depth constructor tool}},
  journal = {Multimedia Tools and Applications},
  year = {2023},
  doi = {10.1007/s11042-023-14743-0},
}
Figure