Introduction
Detecting Human-Object Interaction (HOI) involves identifying triplets of $\langle human, verb, object \rangle$. While essential for understanding complex scenes, scaling HOI detection is hindered by the high cost of manual annotation. Standard fully-supervised models require bounding boxes for every person and object, plus a label for their interaction. In datasets like HICO-DET, which features 600 interaction classes, this process is prohibitively expensive.
The Challenge of Mixed Supervision
Weakly-supervised learning, which uses only image-level tags (e.g., "person-riding-bicycle"), is significantly cheaper but lacks the spatial precision to distinguish which specific entities are interacting. Attempting to combine a small amount of "strong" (fully-labeled) data with a large amount of "weak" data often leads to optimization issues. The gradients from these different supervision levels often conflict, creating a "tug-of-war" that destabilizes training.
The MX-HOI Framework
The authors introduce MX-HOI, a framework designed to leverage mixed supervision effectively through two primary innovations: momentum-independent learning and element swapping.
Momentum-Independent Learning (MIL)
Standard optimizers use a single momentum buffer to smooth updates. However, strong and weak labels present very different error surfaces. MX-HOI maintains two separate momentum buffers—one for weak supervision and one for full supervision. This decoupling prevents the gradients from one source from sabotaging the learning progress of the other.
HOI Element Swapping (HES)
Weakly-supervised models often struggle with "hard negatives"—instances where a person and an object are present but not interacting. To address this, the authors propose HOI Element Swapping. By pairing a human from one image with an object from another, the model is presented with guaranteed negative interactions. This forces the network to learn discriminative features rather than relying on simple co-occurrence statistics.
Results and Impact
The MX-HOI model was evaluated on the HICO-DET benchmark, which contains over 47,000 images and 600 interaction classes.
- Efficiency: Using only 30% fully-labeled data, MX-HOI achieved a mean Average Precision (mAP) of 16.63, which is 93.3% of the performance of a model trained on 100% full supervision (17.82 mAP).
- Generalization: The element swapping technique was particularly beneficial for "Rare" classes (those with fewer than 10 training samples), where data scarcity typically leads to overfitting on co-occurrence.
- Optimization: The momentum-independent strategy allowed the model to outperform existing weakly-supervised and fully-supervised baselines, improving upon the "No-Frills" baseline by approximately 1 mAP point.
Conclusion
MX-HOI demonstrates that high-performance HOI detection does not require exhaustive manual labeling. By addressing the fundamental optimization conflicts in mixed-supervision training and generating synthetic negatives through element swapping, the framework provides a scalable path forward for complex scene understanding.
Summary updated