Detecting Human-Object Interaction with Mixed Supervision

Suresh Kirthi Kumaraswamy, Miaojing Shi, Ewa Kijak

TL;DR

This paper introduces MX-HOI, a framework for Human-Object Interaction (HOI) detection that trains effectively using a mix of expensive, high-quality "strong" labels (bounding boxes) and cheaper, "weak" labels (image-level tags). By using a novel momentum-independent learning strategy and a clever element-swapping technique for data augmentation, the authors achieve 93.3% of the performance of a fully-supervised model while using only 30% of the high-cost annotations.

What Problem Does This Paper Address?

Detecting Human-Object Interactions—identifying a triplet of $\langle human, verb, object \rangle$—is a cornerstone of computer vision. However, it is notoriously difficult to scale. To train a standard model, humans must manually draw bounding boxes around every person and every object and then label the specific action connecting them. In a dataset like HICO-DET, which has 600 interaction classes, this annotation cost is astronomical.

While "weakly-supervised" learning (using only image-level labels like "person-riding-bicycle") is much cheaper, it often lacks the precision to tell which person is riding which bike. Simply mixing strong and weak data usually fails because the gradients (the mathematical directions the model follows to learn) from these two types of data conflict with each other, leading to a "tug-of-war" that degrades performance.

Methods

The authors propose the MX-HOI pipeline, which integrates two key innovations:

  1. Momentum-Independent Learning (MIL): Standard optimizers use "momentum" to smooth out the learning process. However, in mixed supervision, the "error surface" for a weak label is very different from a strong label. Using a single momentum buffer causes the model to oscillate wildly. MX-HOI maintains two separate momentum buffers—one for weak supervision and one for full supervision—allowing the model to learn from both sources without them sabotaging each other.
  2. HOI Element Swapping (HES): A major challenge in weak supervision is "hard negatives"—cases where a person is near an object but not interacting with it. The authors introduce a technique that takes a human from Image A and an object from Image B to create a synthetic "swapped" pair. Since these two were never in the same photo, they are guaranteed to be a negative interaction, forcing the model to learn more discriminative features rather than just relying on the presence of certain objects in a scene.
  3. Two-Branch Architecture: The model uses a classification branch (to identify the verb) and a selection branch (to identify which human-object pair is the most likely candidate), which are multiplied together to produce the final detection.

Datasets

The primary benchmark used is HICO-DET, currently the most challenging dataset for HOI. It contains:

  • 47,776 images total.
  • 600 HOI classes (e.g., "riding a horse," "washing a car").
  • 80 object categories.
  • A split of "Rare" (fewer than 10 training samples) and "Non-Rare" classes to test the model's ability to generalize.

The authors also reference V-COCO as a smaller, standard benchmark in the field.

Key Insights and Ideas

  • Gradient Conflict is the Enemy: The most significant insight is that the failure of mixed supervision isn't just about data quality; it's about optimization. By decoupling the gradient history (momentum), the model can navigate the complex loss landscape of two different supervision levels simultaneously.
  • Synthetic Negatives Matter: In HOI, the model often gets "lazy" and assumes that if a person and a spatula are in the same image, the action must be "cooking." Element swapping breaks this correlation by providing clear examples of "person + object $\neq$ interaction."
  • Efficiency in Labeling: The paper proves that we are hitting a point of diminishing returns with manual annotation. You can get nearly SOTA (State of the Art) results by labeling only a fraction of your dataset thoroughly and the rest lazily.

Key Results

  • The 30/70 Rule: Using only 30% fully-labeled data and 70% weakly-labeled data, MX-HOI achieved a mean Average Precision (mAP) of 16.63. This is remarkably close to the 17.82 mAP achieved using 100% full supervision.
  • Outperforming SOTA: Under the same supervision constraints, MX-HOI outperformed existing weakly-supervised and fully-supervised methods. For instance, it improved upon the "No-Frills" baseline by roughly 1 mAP point across the board.
  • Rare Class Performance: The HOI element swapping was particularly effective for "Rare" classes, where the model lacks enough examples to learn the nuances of the interaction.

Limitations and Future Work

  • Dependency on Object Detectors: Like most HOI frameworks, MX-HOI relies on a pre-trained object detector (like Faster R-CNN). If the detector fails to find the object in the first place, the HOI model has no chance of identifying the interaction.
  • Pseudo-Labeling for Unlabeled Data: While the authors touched on using completely unlabeled data via "pseudo-labels," the performance gains there were more modest compared to the mixed-supervision gains.
  • Future Direction: The authors suggest that future work could focus on developing even stronger weakly-supervised pipelines that can be integrated into this mixed framework, potentially reducing the need for strong labels even further.

Summary updated

BibTeX
@inproceedings{kumaraswamy2021detecting,
  author = {Suresh Kirthi Kumaraswamy and Miaojing Shi and Ewa Kijak},
  title = {{Detecting Human-Object Interaction with Mixed Supervision}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00127},
}
Figure