Improving Pixel-Level Contrastive Learning by Leveraging Exogenous Depth Information

Ahmed Ben Saad, Kristina Prokopetc, Josselin Kherroubi, Axel Davy, Adrien Courtois, Gabriele FaccioloWACV 2023

TL;DR

This paper introduces PixDepth, a novel self-supervised learning method that enhances pixel-level representation learning for tasks like segmentation. By incorporating exogenous depth information into the contrastive learning process, PixDepth significantly improves the accuracy and coherence of learned representations, outperforming existing methods like PixPro.

What Problem Does This Paper Address?

Self-supervised representation learning, particularly through contrastive learning (CL), has shown great promise for tasks like image classification. However, most CL methods are designed for instance-level tasks and struggle to generate representations that are effective for pixel-level tasks such as semantic segmentation. Existing pixel-level CL approaches, like PixPro, rely on 2D image patch distances to define positive and negative pairs. While this is an improvement, it can lead to misclassifications when objects have similar textures or are close in the 2D image but are distinct in 3D space (e.g., due to occlusion or depth differences). This paper addresses the challenge of creating more semantically rich and spatially coherent pixel-level representations by leveraging external depth information to refine the pair selection process.

Methods

The core of PixDepth builds upon the PixPro framework, which uses a contrastive loss on pixel-level features from augmented image crops. The novelty of PixDepth lies in its enhanced pair selection mechanism. Instead of solely relying on 2D spatial distances between image patches, PixDepth incorporates depth information.

Here's a breakdown of the key methods:

  • Exogenous Depth Information: The method leverages depth maps, which can be obtained from various sources like depth estimation networks (e.g., MiDaS), stereo vision, or LiDAR. This depth information is used as an additional filter to distinguish between pixels belonging to different objects.
  • Depth-Based Pair Filtering: For each pair of corresponding pixel patches from two augmented views, PixDepth calculates the absolute difference in their depth values. If this depth difference exceeds a certain threshold, the pair is considered less likely to belong to the same object, thus refining the selection of positive and negative pairs.
  • Multi-Thresholding for Scale Invariance: To address the issue that optimal depth and distance thresholds can vary significantly with object scale across different datasets, PixDepth introduces a multi-threshold loss. This involves dividing the feature map channel-wise into multiple parts and applying different thresholds to each part. The final loss is an average of the losses computed for each part, allowing the representations to be aware of relative context at different scales. This approach reduces the dependency on dataset-specific hyperparameters.

Datasets

The paper uses the following datasets for training and evaluation:

  • ImageNet: Used for pre-training.
  • BoreholeImage: A dataset of ultrasonic borehole images used for geomechanical and geological interpretation. This dataset is used for breakout segmentation.
  • CityScapes: A large-scale dataset of street scenes with pixel-level annotations, used for semantic segmentation.
  • ScanNet: An RGB-D video dataset of indoor scenes with 3D reconstructions and instance-level semantic segmentations, also used for semantic segmentation.

Key Insights and Ideas

  • Depth as a Semantic Cue: The most significant insight is that depth information provides a powerful semantic cue for distinguishing objects, especially in scenarios where 2D appearance alone can be ambiguous. This allows for more accurate positive/negative pair selection in contrastive learning.
  • Simplicity and Availability of Depth: The paper highlights that depth maps are often more readily available than full 3D reconstructions, making PixDepth a more practical approach compared to methods like Pri3D.
  • Multi-Scale Representations: The multi-thresholding strategy is crucial for learning representations that are robust to variations in object scale, a common challenge in real-world datasets. By treating different feature channels with different thresholds, the model can capture finer details and broader contexts simultaneously.
  • Improved Representation Coherence: The qualitative results demonstrate that PixDepth leads to more spatially coherent features, meaning features belonging to the same object are more similar to each other and better contrasted against the background.

Key Results

PixDepth demonstrates significant improvements across various tasks:

  • Breakout Segmentation on Borehole Images: Achieved a 1.9% improvement over PixPro and nearly 5% over the supervised baseline.
  • Indoor Scene Segmentation on CityScapes: Showed a 1.1% improvement over PixPro and over 5% over the supervised baseline.
  • Indoor Scene Segmentation on ScanNet: Achieved a 3.4% improvement over PixPro and 0.6% over Pri3D, with comparable results to Pri3D despite a simpler procedure.
  • Qualitative Improvements: Visualizations show that PixDepth's learned representations are more spatially coherent and better distinguish objects from the background compared to PixPro.

Limitations and Future Work

  • Hyperparameter Sensitivity (Mitigated): While the original PixPro was sensitive to distance thresholds, PixDepth's multi-thresholding approach significantly alleviates this issue. However, the paper notes that using too many thresholds can lead to less expressive features and worse performance.
  • Future Work: The authors suggest exploring PixDepth with more modern backbone architectures, particularly those based on transformers, and investigating its impact on other pixel-level downstream tasks. They also mention that additional experiments with larger models (e.g., ResNet-50) were not feasible within the scope of this study but would be valuable.

Summary updated

BibTeX
@inproceedings{saad2023improving,
  author = {Ahmed Ben Saad and Kristina Prokopetc and Josselin Kherroubi and Axel Davy and Adrien Courtois and Gabriele Facciolo},
  title = {{Improving Pixel-Level Contrastive Learning by Leveraging Exogenous Depth Information}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}

Loading related content...

Figure