Abstract
Human-Object Interaction (HOI) recognition is a fundamental task in visual scene understanding, yet it is plagued by extreme data imbalance and a combinatorial explosion of possible interaction pairs. This paper introduces SGAP-Net, a novel framework that reframes HOI recognition as a few-shot learning problem. By integrating semantic category names (verbs and nouns) into a multi-stage attention mechanism (SIGMA-Module) and employing a "prototype shift" decision method (PS-Module), SGAP-Net effectively identifies complex interactions from minimal examples. Experimental results on the HICO-FS and TUHOI-FS benchmarks demonstrate that SGAP-Net significantly outperforms existing few-shot learning baselines.
The Challenge of HOI Recognition
Recognizing how humans interact with objects (e.g., "riding a horse" vs. "feeding a horse") faces two massive hurdles:
- Extreme Instance Imbalance: Common interactions have thousands of images, while rare ones might have almost none.
- Combinatorial Explosion: There are thousands of possible verb-noun combinations. Training a traditional classifier for every single pair is computationally exhausting and requires an impossible amount of labeled data.
Instead of traditional supervised learning, SGAP-Net utilizes meta-learning (learning to learn), allowing a model to recognize a new interaction after seeing only one or five examples.
Methodology: The SGAP-Net Architecture
The authors introduce the Semantic-Guided Attentive Prototypes Network (SGAP-Net), which consists of two primary innovations designed to leverage semantic priors.
1. SIGMA-Module (SemantIc-Guided Multiple Attention)
Standard attention mechanisms often get "distracted" by the most prominent part of an image (usually the object). To fix this, the SIGMA-Module uses attention-erasing:
- It first uses the noun (object) label to find the object in the image.
- It then "erases" that region and uses the verb (action) label to force the network to find the discriminative regions associated with the action (like hand positions or body posture).
- Finally, it fuses these two views into a single "attentive prototype."
2. PS-Module (Prototypes Shift Module)
In few-shot learning, a "prototype" is the average representation of a class. However, the same action can look very different depending on the person or environment. The PS-Module introduces a Prototype Shift during training. It calculates how much a specific query image "shifts" the class prototype. By using a triplet loss and a regulated shift distance, the model learns a more robust metric space where similar interactions are clustered together despite visual variations.
Experimental Results
Since few-shot HOI is a new formulation, the authors reorganized two standard benchmarks into few-shot versions (HICO-FS and TUHOI-FS) where the test nouns are entirely unseen during training. SGAP-Net was tested against state-of-the-art few-shot algorithms like Prototypical Networks, MAML, and Relation Networks.
On the HICO-FS benchmark:
- 5-way 1-shot: SGAP-Net achieved 38.16% accuracy, outperforming the best baseline (MAML) by over 4.2%.
- 5-way 5-shot: Accuracy jumped to 58.39%, an 11.1% improvement over the nearest competitor.
Ablation studies showed that using both noun and verb guidance with attention-erasing was significantly better than using just one or the other. Specifically, adding the "attention-erasing" step improved 5-way 5-shot performance by nearly 9%.
Conclusion and Future Work
SGAP-Net demonstrates that visual data alone is often insufficient when data is scarce; by leveraging semantic category names as priors, the model can effectively "know" what features to look for. While performance on the more diverse TUHOI-FS dataset was lower, highlighting the difficulty of "in-the-wild" recognition, the semantic-guided nature of the SIGMA-module provides a clear path toward zero-shot HOI recognition. Future work may explore more powerful architectures or vision-language pre-training to further close the performance gap.
Summary updated