Learning Pseudo Front Depth for 2D Forward-Looking Sonar-based Multi-view Stereo

Yusheng Wang, Yonghoon Ji, Hiroshi Tsuchiya, Hajime Asama, Atsushi Yamashita

Abstract

Retrieving the missing dimension information in acoustic images from 2D forward-looking sonar is a well-known problem in the field of underwater robotics. There are works attempting to retrieve 3D information from a single image which allows the robot to generate 3D maps with fly-through motion. However, owing to the unique image formulation principle, estimating 3D information from a single image faces severe ambiguity problems. Classical methods of multi-view stereo can avoid the ambiguity problems, but may require a large number of viewpoints to generate an accurate model. In this work, we propose a novel learning-based multi-view stereo method to estimate 3D information. To better utilize the information from multiple frames, an elevation plane sweeping method is proposed to generate the depth-azimuth-elevation cost volume. The volume after regularization can be considered as a probabilistic volumetric representation of the target. Instead of performing regression on the elevation angles, we use pseudo front depth from the cost volume to represent the 3D information which can avoid the 2D-3D problem in acoustic imaging. High-accuracy results can be generated with only two or three images. Synthetic datasets were generated to simulate various underwater targets. We also built the first real dataset with accurate ground truth in a large scale water tank. Experimental results demonstrate the superiority of our method, compared to other state-of-the-art methods.

Introduction

In murky underwater environments, optical sensors often fail, making 2D Forward-Looking Sonar (FLS) the primary sensing modality for underwater robotics. However, FLS images suffer from a fundamental ambiguity: they collapse 3D space into a 2D plane by losing the elevation angle. A single pixel in a sonar image represents an entire arc in 3D space, leading to the "2D-3D problem." While single-view deep learning methods attempt to resolve this, they often produce "ghostly" or ambiguous reconstructions. Traditional multi-view stereo (MVS) can mitigate this but typically requires an impractical number of viewpoints for real-time robotic applications.

Methodology: Elevation Plane Sweeping Stereo Network (EPSSN)

To address these challenges, we introduce the Elevation Plane Sweeping Stereo Network (EPSSN). This method optimizes 3D reconstruction by adapting the "plane sweeping" technique—common in computer vision—to the specific physics of acoustic imaging.

The EPSSN pipeline: Multi-view sonar frames and relative pose are processed to generate a cost volume for 3D reconstruction.

Elevation Plane Sweeping and Cost Volume

Unlike standard MVS which sweeps planes through depth, EPSSN sweeps virtual planes through elevation angles ($\phi$). By utilizing the known relative pose between a reference frame and a source frame, the network warps source features onto these elevation planes. This process constructs a 4D cost volume (Depth-Azimuth-Elevation) that captures the cross-view consistency.

Pseudo Front Depth Estimation

A key innovation is the use of pseudo front depth. Instead of directly regressing elevation angles for each pixel—a mathematically ill-posed task—we treat the refined cost volume as a probabilistic map. Using a "soft argmax" operation, the network identifies the most likely depth for every point in the volume, effectively bypassing the non-bijective mapping issues inherent in sonar imaging.

Experimental Setup

The model was evaluated using both synthetic and real-world datasets:

  • Synthetic Data: Generated using Blender, including diverse geometries like CAD models, undulating terrains, and spheres to test ambiguity resolution.
  • Real-World Data: A novel dataset collected in a large water tank using an ARIS EXPLORER 3000 sonar. Ground truth was established via land-based photogrammetry of objects before submersion.

Key Results and Discussion

EPSSN demonstrates significant improvements over state-of-the-art single-view and multi-view baselines. In simulated terrain tests, EPSSN achieved a Chamfer Distance (CD) of 0.2200m, vastly outperforming baselines like A2FNet (0.9962m).

Comparison of ground truth and EPSSN-predicted pseudo front depth maps, demonstrating high fidelity in depth estimation.

On real-world data, the method maintained high accuracy (CD of 0.2822m), proving its robustness in practical scenarios. Qualitatively, EPSSN produces clean, recognizable 3D meshes of complex objects, whereas previous methods often failed to capture complete structures.

Conclusion

EPSSN provides a robust solution for 3D underwater mapping using sparse 2D sonar views. By aligning the reconstruction process with sonar physics through elevation plane sweeping and pseudo front depth, it resolves long-standing ambiguities in acoustic imaging. Future work will focus on generalizing the model to arbitrary 6-DOF motions and improving long-range sensing precision.

Summary updated

Abstract

Retrieving the missing dimension information in acoustic images from 2D forward-looking sonar is a well-known problem in the field of underwater robotics. There are works attempting to retrieve 3D information from a single image which allows the robot to generate 3D maps with fly-through motion. However, owing to the unique image formulation principle, estimating 3D information from a single image faces severe ambiguity problems. Classical methods of multi-view stereo can avoid the ambiguity problems, but may require a large number of viewpoints to generate an accurate model. In this work, we propose a novel learning-based multi-view stereo method to estimate 3D information. To better utilize the information from multiple frames, an elevation plane sweeping method is proposed to generate the depth-azimuth-elevation cost volume. The volume after regularization can be considered as a probabilistic volumetric representation of the target. Instead of performing regression on the elevation angles, we use pseudo front depth from the cost volume to represent the 3D information which can avoid the 2D-3D problem in acoustic imaging. High-accuracy results can be generated with only two or three images. Synthetic datasets were generated to simulate various underwater targets. We also built the first real dataset with accurate ground truth in a large scale water tank. Experimental results demonstrate the superiority of our method, compared to other state-of-the-art methods.
BibTeX
@inproceedings{wang2022learning,
  author = {Yusheng Wang and Yonghoon Ji and Hiroshi Tsuchiya and Hajime Asama and Atsushi Yamashita},
  title = {{Learning Pseudo Front Depth for 2D Forward-Looking Sonar-based Multi-view Stereo}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9982049},
}
Figure