Abstract
Detecting hands and the objects they interact with is essential for understanding human activities in video. However, specialized environments like laboratories or factories often suffer from a significant domain gap and a lack of annotated data. While standard Mixup data augmentation is a popular solution for data scarcity, it introduces semantic ambiguity in hand-object contact tasks by creating overlapping "ghost" hands. This paper introduces Background Mixup, a novel augmentation technique that blends training images with empty background frames. This approach preserves semantic clarity while providing the regularization benefits necessary for robust model generalization in data-scarce domains.
Introduction
Accurate detection of hand-object contact is a cornerstone of activity recognition. Despite its importance, training effective detectors is hampered by three primary challenges:
- Domain Gap: Models trained on general datasets (e.g., kitchens) often fail when deployed in specialized settings like biomedical labs.
- Data Scarcity: Expert-level annotation for contact states is expensive and time-consuming, leading to small, easily overfitted datasets.
- Limitations of Standard Mixup: Traditional Mixup overlays two foreground-heavy images, which creates physical impossibilities—such as overlapping hands—that confuse the model's understanding of contact states and object boundaries.
Methodology: Background Mixup
To address the shortcomings of standard Mixup, the authors propose Background Mixup (BG-Mix). The core insight is that for contact detection, adding more foreground entities via augmentation is counterproductive. Instead, the method introduces environmental noise while keeping the "signal" (the hand-object interaction) clean.
The Process
- Background Extraction: A pre-trained detector identifies frames from external sources (like EPIC-KITCHENS-100) that contain no hands or objects-in-contact. These form a "background pool" ($X_{bg}$).
- Blending: A training image ($I_{train}$) is mixed with a randomly sampled background image ($I_{bg}$) using a mixing coefficient $\lambda$: $$\hat{I} = \lambda I_{train} + (1 - \lambda) I_{bg}$$
- Semantic Preservation: Since the background image is empty of foreground targets, the original bounding box labels remain perfectly valid, eliminating the "ghosting" effect seen in standard Mixup.
Experimental Evaluation
The method was tested across several domains, including biomedical experiments and mock factory assembly tasks. Backgrounds were sourced from large-scale datasets like EPIC-KITCHENS-100 and Something-Something V2 to ensure environmental diversity.
Key Results
- Superior Performance: In biomedical lab datasets, BG-Mix achieved 81.0 mAP, outperforming both the supervised baseline (80.7) and standard Mixup (80.6).
- Precision Gains: Standard Mixup often increases recall at the cost of precision by encouraging the model to "hallucinate" boxes. In contrast, BG-Mix improved hand precision to 89.1%, compared to just 76.8% for standard Mixup.
- Data Efficiency: In semi-supervised settings using only 1% of available labels, BG-Mix reached 78.6 mAP, demonstrating its effectiveness for "cold start" scenarios in new domains.
Conclusion
Background Mixup offers a simple yet effective way to regularize hand-object contact detectors without introducing semantic noise. By focusing on "purity over complexity," the method significantly reduces false positives and improves model robustness in specialized, data-scarce environments. Future work may extend this logic to other tasks involving delicate spatial relationships, such as fine-grained tool use or human-robot interaction.
Summary updated