Background Mixup Data Augmentation for Hand and Object-in-Contact Detection

Koya Tango, Takehiko Ohkawa, Ryosuke Furuta, Yoichi SatoarXiv 2022 · arXiv

Abstract

Detecting hands and the objects they interact with is essential for understanding human activities in video. However, specialized environments like laboratories or factories often suffer from a significant domain gap and a lack of annotated data. While standard Mixup data augmentation is a popular solution for data scarcity, it introduces semantic ambiguity in hand-object contact tasks by creating overlapping "ghost" hands. This paper introduces Background Mixup, a novel augmentation technique that blends training images with empty background frames. This approach preserves semantic clarity while providing the regularization benefits necessary for robust model generalization in data-scarce domains.

Introduction

Accurate detection of hand-object contact is a cornerstone of activity recognition. Despite its importance, training effective detectors is hampered by three primary challenges:

  1. Domain Gap: Models trained on general datasets (e.g., kitchens) often fail when deployed in specialized settings like biomedical labs.
  2. Data Scarcity: Expert-level annotation for contact states is expensive and time-consuming, leading to small, easily overfitted datasets.
  3. Limitations of Standard Mixup: Traditional Mixup overlays two foreground-heavy images, which creates physical impossibilities—such as overlapping hands—that confuse the model's understanding of contact states and object boundaries.
Figure 4: Standard Mixup often results in (a) ambiguous contact states and (b) ambiguous object boundaries, making it difficult for models to learn clear physical relationships.

Methodology: Background Mixup

To address the shortcomings of standard Mixup, the authors propose Background Mixup (BG-Mix). The core insight is that for contact detection, adding more foreground entities via augmentation is counterproductive. Instead, the method introduces environmental noise while keeping the "signal" (the hand-object interaction) clean.

The Process

  1. Background Extraction: A pre-trained detector identifies frames from external sources (like EPIC-KITCHENS-100) that contain no hands or objects-in-contact. These form a "background pool" ($X_{bg}$).
  2. Blending: A training image ($I_{train}$) is mixed with a randomly sampled background image ($I_{bg}$) using a mixing coefficient $\lambda$: $$\hat{I} = \lambda I_{train} + (1 - \lambda) I_{bg}$$
  3. Semantic Preservation: Since the background image is empty of foreground targets, the original bounding box labels remain perfectly valid, eliminating the "ghosting" effect seen in standard Mixup.
Figure 6: A comparison between standard Mixup (top), which creates overlapping foregrounds, and Background Mixup (bottom), which preserves the integrity of the original labels.

Experimental Evaluation

The method was tested across several domains, including biomedical experiments and mock factory assembly tasks. Backgrounds were sourced from large-scale datasets like EPIC-KITCHENS-100 and Something-Something V2 to ensure environmental diversity.

Key Results

  • Superior Performance: In biomedical lab datasets, BG-Mix achieved 81.0 mAP, outperforming both the supervised baseline (80.7) and standard Mixup (80.6).
  • Precision Gains: Standard Mixup often increases recall at the cost of precision by encouraging the model to "hallucinate" boxes. In contrast, BG-Mix improved hand precision to 89.1%, compared to just 76.8% for standard Mixup.
  • Data Efficiency: In semi-supervised settings using only 1% of available labels, BG-Mix reached 78.6 mAP, demonstrating its effectiveness for "cold start" scenarios in new domains.

Conclusion

Background Mixup offers a simple yet effective way to regularize hand-object contact detectors without introducing semantic noise. By focusing on "purity over complexity," the method significantly reduces false positives and improves model robustness in specialized, data-scarce environments. Future work may extend this logic to other tasks involving delicate spatial relationships, such as fine-grained tool use or human-robot interaction.

Summary updated

Abstract

Detecting the positions of human hands and objects-in-contact (hand-object detection) in each video frame is vital for understanding human activities from videos. For training an object detector, a method called Mixup, which overlays two training images to mitigate data bias, has been empirically shown to be effective for data augmentation. However, in hand-object detection, mixing two hand-manipulation images produces unintended biases, e.g., the concentration of hands and objects in a specific region degrades the ability of the hand-object detector to identify object boundaries. We propose a data-augmentation method called Background Mixup that leverages data-mixing regularization while reducing the unintended effects in hand-object detection. Instead of mixing two images where a hand and an object in contact appear, we mix a target training image with background images without hands and objects-in-contact extracted from external image sources, and use the mixed images for training the detector. Our experiments demonstrated that the proposed method can effectively reduce false positives and improve the performance of hand-object detection in both supervised and semi-supervised learning settings.
BibTeX
@misc{tango2022background,
  author = {Koya Tango and Takehiko Ohkawa and Ryosuke Furuta and Yoichi Sato},
  title = {{Background Mixup Data Augmentation for Hand and Object-in-Contact Detection}},
  year = {2022},
  eprint = {2202.13941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.13941},
}
Figure