Supervision Interpolation via LossMix: Generalizing Mixup for Object Detection and Beyond

Thanh Vu, Baochen Sun, Bodi Yuan, Alex Ngai, Yueqi Li, Jan-Michael FrahmAAAI 2024 · DOI

TL;DR

This paper introduces LossMix, a novel data augmentation technique that extends the popular Mixup method to object detection. By interpolating loss errors instead of ground truth labels, LossMix effectively addresses the challenges of spatial misalignment and label complexity inherent in object detection, leading to significant performance improvements and setting new state-of-the-art results in both standard object detection and cross-domain object detection tasks.

What Problem Does This Paper Address?

Data augmentation techniques like Mixup have revolutionized image classification by creating synthetic training data through interpolating images and their labels. However, directly applying Mixup to object detection is problematic. Object detection requires precise bounding box coordinates and handles multiple objects with different classes within a single image. Simply interpolating bounding boxes can lead to semantically incorrect results (e.g., a car's bounding box becoming a person's), and interpolating category labels becomes complex due to spatial misalignment and the presence of multiple instances. Existing methods that try to circumvent this by taking a union of bounding boxes don't fully leverage the interpolation principle and can introduce noisy labels, especially when mixing ratios are small. This paper aims to bridge this gap by developing a data mixing strategy that is robust and effective for object detection and other higher-level vision tasks.

Methods

The core contribution is the introduction of Supervision Interpolation (SI), a conceptual framework that generalizes Mixup. Instead of strictly interpolating input images and their corresponding ground truth labels, SI proposes interpolating input images with any form of supervision signal. Building on SI, the authors propose LossMix. The key idea behind LossMix is to interpolate the loss errors of the mixed samples, rather than their labels. This means that instead of creating a new, blended label, the model is trained to minimize a weighted combination of the losses incurred on the original, unmixed samples. This approach implicitly handles the complexities of object detection, such as spatial misalignment and multiple object instances, by re-weighting the penalty for each prediction based on its augmented visibility. LossMix is designed to be loss-agnostic, meaning it can be applied to various loss functions used in object detection (e.g., classification and regression losses).

Datasets

The paper evaluates LossMix on two standard object detection benchmarks:

  • PASCAL VOC (2010/2007): Used for training and evaluation, comprising 16,551 images with 20 categories.
  • MS COCO (2014): A larger dataset with 80 object categories, used for further validation.

For domain adaptation experiments, the authors use:

  • PASCAL VOC → Clipart1k: A cross-domain adaptation task.
  • PASCAL VOC → Watercolor2k: Another cross-domain adaptation task, representing a larger domain shift.

Key Insights and Ideas

  • Loss Interpolation over Label Interpolation: The most significant insight is that interpolating loss errors is a more robust and generalizable approach for complex tasks like object detection than interpolating ground truth labels. This bypasses the semantic inconsistencies that arise from directly mixing bounding boxes and category labels.
  • Flexibility of Supervision Interpolation: The SI framework is a powerful generalization. It shows that the core principle of Mixup (interpolating inputs and supervision) can be applied to various forms of supervision, not just hard labels, opening doors for new augmentation strategies.
  • Implicit Handling of Object Detection Challenges: LossMix naturally handles spatial misalignment and label complexity by re-weighting the loss based on the interpolation coefficient. This means that objects with less visibility in the mixed image contribute less to the overall loss, preventing the model from being overly penalized for detecting partially obscured or less prominent objects.
  • Versatility Beyond Detection: The paper demonstrates that LossMix is not limited to object detection and can enhance performance in semi-supervised learning and unsupervised domain adaptation tasks.

Key Results

  • Object Detection Performance: LossMix consistently outperforms baseline models and existing mixing strategies like "Union" and "Noise" on both PASCAL VOC and MS COCO datasets. For example, on PASCAL VOC, LossMix achieved up to +0.9 AP over Union and +1.5 AP over Noise.
  • State-of-the-Art in Domain Adaptation: By integrating LossMix with the Adaptive Teacher framework, the authors achieve new state-of-the-art results on cross-domain object detection tasks, notably outperforming previous methods on PASCAL VOC → Clipart1k and PASCAL VOC → Watercolor2k.
  • Robustness: Ablation studies show that LossMix is robust to different configurations and hyperparameter choices, consistently providing improvements over baselines.

Limitations and Future Work

  • Hyperparameter Tuning: While LossMix is generally robust, specific hyperparameter choices (e.g., the mixing coefficient $\lambda$) can still influence performance, as seen in the ablation studies.
  • Future Work: The authors suggest that LossMix's generalizability opens doors for more applications beyond object detection, potentially in other dense prediction tasks like segmentation or depth estimation. They also highlight the potential for exploring different input mixing strategies in conjunction with LossMix.

Summary updated

BibTeX
@inproceedings{vu2024supervision,
  author = {Thanh Vu and Baochen Sun and Bodi Yuan and Alex Ngai and Yueqi Li and Jan-Michael Frahm},
  title = {{Supervision Interpolation via LossMix: Generalizing Mixup for Object Detection and Beyond}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28335},
}
Figure