SGAP-Net: Semantic-Guided Attentive Prototypes Network for Few-Shot Human-Object Interaction Recognition

Zhong Ji, Xiyao Liu, Yanwei Pang, Xuelong Li

Abstract

Human-Object Interaction (HOI) recognition is a fundamental task in visual scene understanding, yet it is plagued by extreme data imbalance and a combinatorial explosion of possible interaction pairs. This paper introduces SGAP-Net, a novel framework that reframes HOI recognition as a few-shot learning problem. By integrating semantic category names (verbs and nouns) into a multi-stage attention mechanism (SIGMA-Module) and employing a "prototype shift" decision method (PS-Module), SGAP-Net effectively identifies complex interactions from minimal examples. Experimental results on the HICO-FS and TUHOI-FS benchmarks demonstrate that SGAP-Net significantly outperforms existing few-shot learning baselines.

The Challenge of HOI Recognition

Recognizing how humans interact with objects (e.g., "riding a horse" vs. "feeding a horse") faces two massive hurdles:

  1. Extreme Instance Imbalance: Common interactions have thousands of images, while rare ones might have almost none.
  2. Combinatorial Explosion: There are thousands of possible verb-noun combinations. Training a traditional classifier for every single pair is computationally exhausting and requires an impossible amount of labeled data.

Instead of traditional supervised learning, SGAP-Net utilizes meta-learning (learning to learn), allowing a model to recognize a new interaction after seeing only one or five examples.

Methodology: The SGAP-Net Architecture

The authors introduce the Semantic-Guided Attentive Prototypes Network (SGAP-Net), which consists of two primary innovations designed to leverage semantic priors.

Overview of the SGAP-Net architecture, showing the flow from the support set through the SIGMA and PS modules to the final query classification.

1. SIGMA-Module (SemantIc-Guided Multiple Attention)

Standard attention mechanisms often get "distracted" by the most prominent part of an image (usually the object). To fix this, the SIGMA-Module uses attention-erasing:

  • It first uses the noun (object) label to find the object in the image.
  • It then "erases" that region and uses the verb (action) label to force the network to find the discriminative regions associated with the action (like hand positions or body posture).
  • Finally, it fuses these two views into a single "attentive prototype."

2. PS-Module (Prototypes Shift Module)

In few-shot learning, a "prototype" is the average representation of a class. However, the same action can look very different depending on the person or environment. The PS-Module introduces a Prototype Shift during training. It calculates how much a specific query image "shifts" the class prototype. By using a triplet loss and a regulated shift distance, the model learns a more robust metric space where similar interactions are clustered together despite visual variations.

Experimental Results

Since few-shot HOI is a new formulation, the authors reorganized two standard benchmarks into few-shot versions (HICO-FS and TUHOI-FS) where the test nouns are entirely unseen during training. SGAP-Net was tested against state-of-the-art few-shot algorithms like Prototypical Networks, MAML, and Relation Networks.

Accuracy trends for SGAP-Net on the 5-way HICO-FS experiment set, showing consistent performance gains from 1-shot to 5-shot scenarios.

On the HICO-FS benchmark:

  • 5-way 1-shot: SGAP-Net achieved 38.16% accuracy, outperforming the best baseline (MAML) by over 4.2%.
  • 5-way 5-shot: Accuracy jumped to 58.39%, an 11.1% improvement over the nearest competitor.

Ablation studies showed that using both noun and verb guidance with attention-erasing was significantly better than using just one or the other. Specifically, adding the "attention-erasing" step improved 5-way 5-shot performance by nearly 9%.

Conclusion and Future Work

SGAP-Net demonstrates that visual data alone is often insufficient when data is scarce; by leveraging semantic category names as priors, the model can effectively "know" what features to look for. While performance on the more diverse TUHOI-FS dataset was lower, highlighting the difficulty of "in-the-wild" recognition, the semantic-guided nature of the SIGMA-module provides a clear path toward zero-shot HOI recognition. Future work may explore more powerful architectures or vision-language pre-training to further close the performance gap.

Summary updated

BibTeX
@inproceedings{ji2020sgap,
  author = {Zhong Ji and Xiyao Liu and Yanwei Pang and Xuelong Li},
  title = {{SGAP-Net: Semantic-Guided Attentive Prototypes Network for Few-Shot Human-Object Interaction Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6764},
}
Figure