TL;DR
MARTINE is a new high-fidelity benchmark for 3D surface reconstruction that provides ultra-high-resolution imagery (61MP) across 28 diverse objects and two distinct lighting regimes. By offering precise 3D ground truth and "challenge-level" masks, it allows researchers to pinpoint exactly how algorithms struggle with specific materials like glass, fine structures, or specular highlights.
What Problem Does This Paper Address?
Benchmarking multi-view 3D reconstruction is notoriously difficult because performance is a "moving target" that depends on the interaction between the algorithm, the object's material properties, and the capture conditions (resolution, lighting, and viewpoint coverage).
Existing datasets often have significant gaps: some lack high-resolution images, others have limited viewpoint variety, and many suffer from poor "image-to-geometry" alignment. If the 3D ground-truth scan isn't perfectly aligned with the 2D images, the resulting error metrics are misleading. Furthermore, most benchmarks provide a single aggregate score per object, which hides why a method failed—was it the transparency of the glass or the thinness of a wire? MARTINE addresses these issues by providing a "regime-aware" evaluation framework with sub-millimetric accuracy.
Methods
The authors built a sophisticated acquisition rig using an industrial robotic arm and a motorized turntable to capture 84 views over a spherical cap.
Key technical innovations include:
- Dual Lighting Regimes: Each object is captured under two setups: a single-light ambient regime (using a reflecting cone and softbox) for standard photogrammetry, and a multi-light regime (using a rig of 30 LEDs) to support photometric stereo methods.
- Object-Agnostic Calibration: Instead of calibrating based on the object itself (which can introduce bias), they use Structure-from-Motion (SfM) on the textured black cloth of the turntable, achieving extremely low reprojection errors (~0.04 mm).
- Contour-Based Alignment: To ensure the 3D ground truth matches the images, they developed a contour-based Iterative Closest Point (ICP) method that aligns silhouettes, reaching a median accuracy of 1.2 pixels at 61MP resolution.
- Challenge-Level Masks: They manually annotated 3D masks for properties like "transparency," "fine structure," and "high curvature," enabling a granular analysis of reconstruction errors.
Datasets
The paper introduces the MARTINE dataset, which consists of:
- 28 objects featuring diverse materials (matte, specular, transparent) and geometries (fine structures, deep concavities).
- 84 viewpoints per object at 9568 × 6376 resolution.
- Ground-truth 3D scans (mostly from an Artec Spider optical scanner, with some CT scans for complex internal structures).
- Comparison with existing benchmarks: The authors contrast MARTINE against standard sets like DTU, DiLiGenT-MV, and OpenSubstance, highlighting MARTINE's unique combination of ultra-high resolution and quantified alignment accuracy.
Key Insights and Ideas
- Alignment is the "Silent Killer": The authors demonstrate that even high-quality datasets like OpenSubstance can have alignment errors that are poorly captured by standard metrics like IoU. MARTINE's focus on quantified 2D-3D registration is a major step forward for metric evaluation.
- Resolution is a Double-Edged Sword: While high resolution provides more matching features for photogrammetry, it also exposes the limitations of current neural implicit methods, many of which fail to run at full resolution due to GPU memory constraints.
- The Photometric Gap: Surprisingly, multi-light photometric cues didn't automatically lead to higher global accuracy in current state-of-the-art implementations. This suggests that better fusion strategies are needed to fully exploit shading information in multi-view pipelines.
- Granular Failure Analysis: By using property masks, the authors show that a method might have a great "average" score but fail completely on fine structures or low-visibility regions.
Key Results
The authors evaluated several paradigms, including classical MVS (COLMAP), neural implicit surfaces (NeuS2, Neuralangelo), and Gaussian Splatting (PGSR).
- Top Performers: Neural implicit methods reached the highest F-scores (up to 77.9% at a 0.5 mm threshold).
- Precision vs. Recall: COLMAP showed the highest precision (accuracy of reconstructed points) but lower recall (completeness) compared to neural methods.
- Sub-millimetric Resolution: The benchmark successfully resolved differences as small as 0.058 mm (Chamfer distance) for the best-performing methods.
- Resolution Impact: Photogrammetry pipelines like RealityScan saw massive performance drops (from 99.3% to 28.0% F-score) when downsampling from full to 1/8th resolution, proving the value of MARTINE's high-res data.
Limitations and Future Work
- Scale: The dataset is currently limited to 28 objects due to the intensive acquisition process (roughly one day per object).
- Manual Labeling: The property masks are currently segmented manually; the authors suggest that future work could automate this using objective geometric criteria.
- Task Scope: While the data is suitable for tasks like novel-view synthesis and inverse rendering, this initial paper focuses strictly on surface geometry reconstruction.
Summary updated