TW-SMNet: Deep Multitask Learning of Tele-Wide Stereo Matching

Mostafa El-Khamy, Haoyu Ren, Xianzhi Du, Jungwon LeearXiv 2019 · arXiv

Introduction

Modern smartphones increasingly rely on multi-camera systems to enhance photographic capabilities. A common configuration is the "Tele-Wide" setup, featuring one wide-angle lens and one telephoto lens. While this pairing offers versatility, it presents a significant challenge for depth estimation: the telephoto lens only captures the central portion of the wide-angle camera's field of view (FOV). Researchers at Samsung Semiconductor have developed TW-SMNet, a deep multitask learning framework designed to generate high-quality, full-frame depth maps from these heterogeneous camera pairs.

The Tele-Wide Challenge

Standard stereo matching algorithms typically assume two identical cameras separated by a small baseline. Depth is calculated by measuring the horizontal disparity of objects between the two viewpoints. In a Tele-Wide setup, this process is complicated by the mismatch in FOV and resolution.

The telephoto lens (indicated by the red box) captures only the central region of the wide-angle camera's field of view.

As shown in the figure above, the scene is divided into two distinct areas:

  1. The Overlap Region: The central area visible to both cameras, allowing for high-accuracy stereo matching.
  2. The Surrounding Region: The peripheral edges visible only to the wide-angle camera. In this region, the system is effectively "blind" in one eye, precluding traditional stereo techniques.

Proposed Architecture: TW-SMNet

The authors introduce a hierarchy of solutions to address this disparity. The baseline TW-SMNet adapts the PSMNet architecture, utilizing 3D convolutions and a "stacked hourglass" design to estimate disparity. To handle the tele-wide geometry, images are zero-padded or cropped to align resolutions.

The core innovation, however, is the MT-TW-SMNet (Multitask) architecture. This model employs a shared feature extractor (ResNet-50 with Spatial Pyramid Pooling) that branches into two specialized tasks:

  • Stereo Matching Disparity Estimation (SMDE): Focuses on the overlapping central region to leverage precise geometric information.
  • Single Image inverse Depth Estimation (SIDE): Operates on the full wide FOV, using monocular cues and scene understanding (e.g., the convergence of road lines) to "hallucinate" depth in the peripheral regions.

Multitask Learning and Fusion

A key insight of this research is that multitask training provides a synergistic effect. Training the network for monocular depth estimation actually improves its stereo matching performance. The monocular task forces the feature extractor to grasp global scene context, which helps the stereo branch resolve ambiguities in challenging areas.

To create a seamless depth map, the authors explored several fusion strategies:

  • Input Feature Fusion: Using stereo output as a sparse depth guide for the monocular network.
  • Output Fusion: Selecting stereo results for the center and monocular results for the edges, followed by a Fast Global Smoother (FGS) to eliminate visible seams.
  • Deep Network Fusion: A dedicated sub-network learned to blend the depth maps optimally.

Results and Applications

The MT-TW-SMNet was evaluated on the SceneFlow and KITTI Stereo 2015 benchmarks. On the KITTI test set, the "MT-TW Fusion" model achieved a full-FOV disparity error of 11.96%, outperforming several methods that require full-frame stereo pairs.

A synthesized wide-angle Bokeh effect using the depth map generated by TW-SMNet, showing natural background blur across the entire frame.

The practical utility of this approach is most evident in the synthesis of "Wide FOV Bokeh" effects. By providing an accurate depth map for the entire wide-angle frame, the system enables smooth, natural-looking background blur that transitions correctly from the high-precision center to the monocularly-estimated edges.

Conclusion

TW-SMNet successfully bridges the gap between stereo matching and monocular depth estimation for modern smartphone camera arrays. By leveraging multitask learning and intelligent fusion, the system maintains sub-pixel accuracy in the center while intelligently filling in the edges. Future work may focus on optimizing these computationally intensive 3D convolutions for real-time performance on mobile hardware and extending the logic to triple-camera configurations.

Summary updated

BibTeX
@misc{elkhamy2019tw,
  author = {Mostafa El-Khamy and Haoyu Ren and Xianzhi Du and Jungwon Lee},
  title = {{TW-SMNet: Deep Multitask Learning of Tele-Wide Stereo Matching}},
  year = {2019},
  eprint = {1906.04463},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.04463},
}

Loading related content...

Figure