Learning Pseudo Front Depth for 2D Forward-Looking Sonar-based Multi-view Stereo

Yusheng Wang, Yonghoon Ji, Hiroshi Tsuchiya, Hajime Asama, Atsushi Yamashita

TL;DR

Researchers have developed the Elevation Plane Sweeping Stereo Network (EPSSN), a deep learning method that reconstructs dense 3D underwater scenes from just two or three 2D sonar images. By adapting traditional "plane sweeping" techniques to the unique geometry of acoustic cameras, the system resolves the inherent ambiguity of sonar data to produce high-accuracy 3D maps where previous methods failed.

What Problem Does This Paper Address?

In murky underwater environments, optical cameras are useless. Robots rely on 2D Forward-Looking Sonar (FLS), which acts like an "acoustic camera." However, FLS has a major mathematical flaw: it collapses 3D space into a 2D image by losing the elevation angle.

Imagine a single pixel in a sonar image; it doesn't represent a single point in space, but rather an entire arc of possible locations. This is known as the 2D-3D problem. While some AI models try to guess the 3D shape from a single image, the results are often ambiguous or "ghostly" because one pixel could represent multiple surfaces. Traditional multi-view stereo (MVS) can solve this but usually requires dozens of images from different angles, which is impractical for a moving underwater drone.

Methods

The authors propose EPSSN, which treats 3D reconstruction as a multi-view stereo problem but optimizes it for the specific physics of sound.

  1. Elevation Plane Sweeping: In computer vision, "plane sweeping" usually involves moving a virtual plane through depth. Here, the authors sweep virtual planes through elevation angles ($\phi$).
  2. Cost Volume Construction: They take a "reference" sonar frame and a "source" frame. Using the known movement of the robot (relative pose), they warp the features of the source frame onto the elevation planes of the reference frame. This creates a 4D "cost volume" (Depth-Azimuth-Elevation) that represents how well the images match at different 3D coordinates.
  3. Pseudo Front Depth: Instead of trying to predict an elevation angle for every pixel (which is mathematically messy), they calculate a "pseudo front depth." They treat the refined cost volume as a probabilistic map and use a "soft argmax" to find the most likely depth for every point in the 3D volume.
  4. 3D CNN Refinement: A 3D U-Net-style architecture is used to clean up the noisy initial cost volume, helping the network understand the global structure of the scene.

Datasets

The study utilized both simulated and real-world data:

  • Simulation (Blender): The authors created three synthetic datasets: "Artificial" (CAD models), "Terrain" (undulating seafloors), and "Sphere" (specifically designed to test ambiguity). They generated 4,000 image pairs for training.
  • Real-World Dataset: This is a significant contribution—the first large-scale real sonar dataset with accurate ground truth. They used an ARIS EXPLORER 3000 sonar in a massive water tank. Ground truth was captured using land-based photogrammetry of the objects before they were submerged.

Key Insights and Ideas

  • Geometry Adaptation: The "aha moment" is realizing that sonar MVS shouldn't sweep through depth like a camera, but through elevation. This aligns the neural network's processing with the actual physics of how sonar images are formed.
  • Solving Ambiguity with Two Views: The paper demonstrates that while a single image is mathematically "ill-posed" (impossible to solve perfectly), adding just one more viewpoint with a small "roll" rotation (5–10°) provides enough geometric constraints to resolve most 3D ambiguities.
  • Pseudo Depth vs. Elevation Regression: By predicting depth within the volume rather than an angle for a pixel, the model avoids the "non-bijective" trap where one pixel maps to multiple points.

Key Results

EPSSN significantly outperformed state-of-the-art single-view and multi-view baselines (like A2FNet and ElevateNet):

  • Accuracy: In simulated terrain tests, EPSSN achieved a Chamfer Distance (CD) of 0.2200m, compared to A2FNet’s 0.9962m.
  • Real-World Performance: On the water tank dataset, EPSSN maintained a CD of 0.2822m, while the best baseline was nearly three times less accurate (0.7605m).
  • Visual Quality: Qualitative results show that EPSSN produces clean, recognizable 3D meshes of complex objects (like buckets or industrial parts), whereas previous methods often produced distorted or incomplete shapes.

Limitations and Future Work

  • Motion Constraints: The current experiments focused on simple "roll" rotations. Real-world underwater drones move in six degrees of freedom. The authors note that future work must test the robustness of the model against more complex, arbitrary motions.
  • Sim-to-Real Gap: While the model performed well on real data, it still benefits from being fine-tuned on synthetic data. The authors suggest exploring Generative Adversarial Networks (GANs) or self-supervised learning to make the model more adaptable to different oceanic conditions without needing labeled ground truth.
  • Range Precision: The method was tested at relatively short ranges (approx. 2 meters). Its precision at long-range sensing remains an open question.

Summary updated

Abstract

Retrieving the missing dimension information in acoustic images from 2D forward-looking sonar is a well-known problem in the field of underwater robotics. There are works attempting to retrieve 3D information from a single image which allows the robot to generate 3D maps with fly-through motion. However, owing to the unique image formulation principle, estimating 3D information from a single image faces severe ambiguity problems. Classical methods of multi-view stereo can avoid the ambiguity problems, but may require a large number of viewpoints to generate an accurate model. In this work, we propose a novel learning-based multi-view stereo method to estimate 3D information. To better utilize the information from multiple frames, an elevation plane sweeping method is proposed to generate the depth-azimuth-elevation cost volume. The volume after regularization can be considered as a probabilistic volumetric representation of the target. Instead of performing regression on the elevation angles, we use pseudo front depth from the cost volume to represent the 3D information which can avoid the 2D-3D problem in acoustic imaging. High-accuracy results can be generated with only two or three images. Synthetic datasets were generated to simulate various underwater targets. We also built the first real dataset with accurate ground truth in a large scale water tank. Experimental results demonstrate the superiority of our method, compared to other state-of-the-art methods.
BibTeX
@inproceedings{wang2022learning,
  author = {Yusheng Wang and Yonghoon Ji and Hiroshi Tsuchiya and Hajime Asama and Atsushi Yamashita},
  title = {{Learning Pseudo Front Depth for 2D Forward-Looking Sonar-based Multi-view Stereo}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9982049},
}

Loading related content...

Figure