Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection

Deepti Hegde, Vishal M. Patel

TL;DR

This paper introduces "Attentive Prototypes," a novel source-free domain adaptation method for 3D object detection. It uses a transformer to learn robust class prototypes from noisy pseudo-labels, effectively mitigating performance drops when models encounter data from different environments or sensors, outperforming existing methods.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on 3D object detection to perceive their surroundings. However, deep learning models trained on specific datasets (e.g., from one city or sensor) often perform poorly when deployed in new environments with different conditions, sensors, or data distributions. This "domain shift" is a major challenge. Traditional domain adaptation techniques often require access to the original training data (source data), which might be unavailable due to privacy or memory constraints. Furthermore, existing source-free methods either need sequences of lidar frames or struggle with noisy pseudo-labels generated during self-training, where incorrect predictions can reinforce errors. This paper tackles the problem of adapting 3D object detectors to new domains without access to source data and using only single lidar frames, specifically by addressing the issue of noisy pseudo-labels.

Methods

The core of this work is the "attentive prototype" mechanism. Instead of simply averaging region features to create a class prototype (which can be corrupted by noisy labels), they employ a transformer module. This transformer uses self-attention to identify "salient regions-of-interest" within the detected objects. These attentive features are then combined using "predictive entropy weights," which reflect the model's uncertainty about each prediction. This allows the method to down-weight features from potentially mislabeled or overconfident regions.

The adaptation process is iterative:

  1. Pseudo-label Generation: The source-trained model generates initial pseudo-labels on the target domain data.
  2. Attentive Prototype Computation: A transformer module processes region features from these pseudo-labels to compute an attentive class prototype.
  3. Loss Weighting: The similarity of each region's features to the computed prototype is used to create "similarity weights." These weights are then used to down-weight the classification loss for regions that are dissimilar to the prototype, effectively filtering out noisy pseudo-labels.
  4. Model Retraining: The object detector is retrained using these weighted losses.
  5. Iteration: This process repeats, with the model becoming progressively better at generating accurate pseudo-labels and refining its prototypes.

An exponential moving average (EMA) is used to combine prototypes across iterations, ensuring a stable learning process.

Datasets

The experiments were conducted using three large-scale autonomous driving datasets:

  • Waymo Open Dataset: A large dataset with significant domain gaps.
  • KITTI Dataset: A smaller, well-established dataset.
  • nuScenes Dataset: Another large, multimodal dataset.

The paper explores three cross-dataset domain shift scenarios for the "Car" object category: Waymo → KITTI, nuScenes → KITTI, and KITTI → nuScenes. They also evaluate Waymo → nuScenes.

Key Insights and Ideas

  • Transformer for Robust Prototypes: The key insight is that self-attention mechanisms, commonly used for understanding relationships in sequences, can effectively identify important features for prototype computation, even when labels are noisy. This is a significant improvement over simple averaging.
  • Predictive Entropy as a Confidence Measure: Using predictive entropy to weight features provides a principled way to incorporate model uncertainty into prototype generation, further helping to filter out unreliable predictions.
  • Iterative Refinement of Pseudo-Labels: The iterative training strategy, combined with prototype-based loss weighting, allows the model to progressively refine its understanding of the target domain and improve the quality of its pseudo-labels over time.
  • Source-Free and Single-Frame Adaptation: Achieving effective domain adaptation without access to source data and using only single frames is a crucial step towards practical deployment in real-world scenarios where data access is limited.

Key Results

The proposed "Attentive Prototypes" method consistently outperforms existing domain adaptation techniques across various domain shift scenarios and for two different 3D object detection architectures (SECOND-iou and PointRCNN).

  • For the SECOND-iou detector, the proposed method achieved a 3D mAP of 75.75 on Waymo → KITTI (easy), significantly outperforming direct transfer (46.66), ST (69.04), ST3D (67.47), and SN (73.23).
  • In the nuScenes → KITTI scenario with SECOND-iou, the method reached 71.56 (easy), again surpassing other methods.
  • For PointRCNN, the proposed method achieved 62.11 on Waymo → KITTI (easy), also showing strong improvements.
  • Qualitative results (Figure 5) show that the proposed method produces more accurate bounding boxes with fewer false positives compared to direct transfer and other methods.

Limitations and Future Work

  • Single Class Prototype: The current method computes a single attentive prototype per class. The authors acknowledge that for detecting smaller objects like pedestrians and cyclists, which are less frequent and often exhibit larger domain shifts, extending the method to learn multiple prototypes per class using class-specific transformer modules would be beneficial.
  • Closed-Set Domain Adaptation: The current work assumes that the classes present in the source domain are also present in the target domain. This "closed-set" assumption might not hold in all real-world scenarios.
  • Dataset Size Shift: The method is less effective when addressing shifts from smaller to larger datasets (e.g., KITTI to nuScenes).
  • Future Work: The authors suggest exploring fully unsupervised solutions for domain shifts that involve differences in object sizes (which statistical normalization partially addresses) and extending the framework to handle open-set domain adaptation.

Summary updated

Abstract

3D object detection networks tend to be biased towards the data they are trained on. Evaluation on datasets captured in different locations, conditions or sensors than that of the training (source) data results in a drop in model performance due to the gap in distribution with the test (or target) data. Current methods for domain adaptation either assume access to source data during training, which may not be available due to privacy or memory concerns, or require a sequence of lidar frames as an input. We propose a single-frame approach for source-free, unsupervised domain adaptation of lidar-based 3D object detectors that uses class prototypes to mitigate the effect pseudo-label noise. Addressing the limitations of traditional feature aggregation methods for prototype computation in the presence of noisy labels, we utilize a transformer module to identify outlier ROI's that correspond to incorrect, over-confident annotations, and compute an attentive class prototype. Under an iterative training strategy, the losses associated with noisy pseudo labels are down-weighed and thus refined in the process of self-training. To validate the effectiveness of our proposed approach, we examine the domain shift associated with networks trained on large, label-rich datasets (such as the Waymo Open Dataset and nuScenes) and evaluate on smaller, label-poor datasets (such as KITTI) and vice-versa. We demonstrate our approach on two recent object detectors and achieve results that out-perform the other domain adaptation works.
BibTeX
@inproceedings{hegde2024attentive,
  author = {Deepti Hegde and Vishal M. Patel},
  title = {{Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure