Multiview Co-segmentation for Wide Baseline Images using Cross-view Supervision

Yuan Yao, Hyun Soo ParkWACV 2020

TL;DR

This paper introduces a novel semi-supervised learning framework for co-segmenting objects in wide-baseline multiview images, where traditional photometric matching fails. Their key innovation is a differentiable "shape belief transfer" mechanism that leverages epipolar geometry to propagate segmentation information across views, enabling end-to-end training without relying on extensive manual annotations.

What Problem Does This Paper Address?

Co-segmentation, the task of identifying and outlining common objects across multiple images, is challenging. This paper specifically tackles the problem in the context of "wide-baseline" images. Wide-baseline scenarios arise when cameras are positioned far apart or at significantly different angles, making it difficult to find corresponding pixels between images using standard photometric matching techniques. This situation is common in real-world applications like capturing social videos from multiple mobile devices. Existing methods often struggle with the fragility of local features, the complexity of graph matching in the presence of self-occlusions, or are limited to pre-defined object classes. The authors aim to overcome these limitations by enabling co-segmentation for novel object classes in challenging wide-baseline settings with minimal labeled data.

Methods

The core of the proposed method is a novel "shape belief transfer" mechanism. Inspired by shape-from-silhouette techniques, which reconstruct 3D shapes from 2D segmentation masks without needing photometric matching, the authors formulate a way to transfer segmentation "belief" (confidence) from one view to another. This is achieved by:

  1. 3D Shape Reconstruction (Implicit): They first reconstruct a 3D shape representation by triangulating segmentation confidences from multiple source views.
  2. 2D Projection: This 3D shape is then projected back onto a target view to approximate its segmentation mask.

Crucially, this entire process is made differentiable, allowing for end-to-end training of a neural network. They introduce a "triplet network" architecture that takes multiple views as input and outputs object confidence maps. The network is trained by minimizing three losses:

  • Labeled Supervision Loss ($L_L$): Standard cross-entropy loss on the limited labeled data.
  • Cross-view Supervision Loss ($L_S$): A novel one-way relative cross-entropy loss that penalizes inconsistencies between the predicted segmentation in the target view and the shape belief transferred from source views. This loss is key to leveraging unlabeled data.
  • Bootstrapping Prior Loss ($L_P$): A loss used for unlabeled data, which leverages the binarized segmentation from source views to create a pseudo-label for the target view.

To handle the computational challenges of projecting along epipolar lines, they employ a stereo rectification technique to convert oblique line max-pooling into regular row-wise max-pooling, simplifying the operation.

Datasets

The authors evaluate their framework on several real-world datasets:

  • Monkey Subject: Captured by 35 GoPro HD cameras in a large cage.
  • Indian Dancer: Recorded by 69 synchronized HD cameras in a multi-layer setup.
  • Social Videos: A public dataset of break dancers surrounded by audiences, captured by GoPro cameras.

These datasets are characterized by wide-baseline camera configurations and dynamic scenes, making them challenging for existing methods. They uniformly sample frames and annotate a small subset of views as labeled data, using the rest as unlabeled data for training.

Key Insights and Ideas

  • Shape Belief Transfer as a Differentiable Operation: The most significant insight is framing the transfer of segmentation information across views as a differentiable process. This bypasses the need for explicit 3D reconstruction and enables end-to-end learning, a major step forward for semi-supervised co-segmentation in wide-baseline settings.
  • Leveraging Epipolar Geometry for Supervision: The paper elegantly uses epipolar geometry to connect pixels across views, forming the basis for the cross-view supervision signal. This is a geometrically sound approach that doesn't rely on fragile photometric correspondences.
  • Upper and Lower Bounds for Unlabeled Data: The theoretical analysis of shape belief transfer reveals that it provides both upper and lower bounds for the segmentation of unlabeled data. This theoretical grounding helps understand the convergence properties and limitations of the method.
  • Flexibility of the Framework: The proposed method is flexible, allowing customization of the segmentation backbone and applicability to various multi-camera systems, including challenging social video scenarios.

Key Results

The proposed method consistently outperforms several baseline algorithms (supervised learning, bootstrapping, attention-based co-segmentation, and adversarial segmentation) across different datasets and metrics (mean IoU and pixel accuracy). For instance, on the Monkey dataset with 8 labeled views, their method achieves a mean IoU of 0.87, significantly higher than the best baseline at 0.82. They also demonstrate that their approach is more robust to label data sensitivity (temporal distance between labeled and unlabeled data) compared to other semi-supervised methods. Qualitative results show their framework effectively handles wide-baseline images and corrects segmentation errors, particularly on object boundaries.

Limitations and Future Work

  • Degenerate Case: The paper acknowledges a degenerate case where a trivial solution ($P_t = 0$) can be a global minimizer, potentially limiting the effectiveness of cross-view supervision when unlabeled data is used as the target view.
  • Computational Cost: While end-to-end training is enabled, the process of shape belief transfer and rectification might still be computationally intensive.
  • Future Work: The authors suggest further exploration of the theoretical bounds and their implications. Additionally, extending the framework to handle more complex scenarios with dynamic backgrounds and a larger number of views could be a promising direction. The paper also hints at the potential for this method to improve existing co-segmentation techniques by correcting erroneous manual annotations.

Summary updated

BibTeX
@inproceedings{yao2020multiview,
  author = {Yuan Yao and Hyun Soo Park},
  title = {{Multiview Co-segmentation for Wide Baseline Images using Cross-view Supervision}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure