Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning

Chengxiang Yin, Kun Wu, Zhengping Che, Bo Jiang, Zhiyuan Xu, Jian Tang

Abstract

The Hierarchical Graph Attention Network (HGAT) is a novel two-stage framework designed to solve multi-modal tasks like Visual Question Answering (VQA) using only a handful of examples. By explicitly modeling both the relationships within a single modality and the "co-learning" cross-talk between images and text, HGAT achieves state-of-the-art performance in few-shot and semi-supervised settings.

Introduction

Deep learning typically requires massive datasets to function effectively. However, humans possess the remarkable ability to look at a single photo and a related snippet of text to immediately understand the context—a capability known as few-shot visual-semantic learning. This remains a significant hurdle for artificial intelligence.

Existing approaches often attempt to adapt general meta-learning algorithms, such as MAML, to multi-modal problems. However, these methods frequently treat visual and text data as flat features, failing to capture the intricate "intra-modal" (relationships within images) and "inter-modal" (how a caption clarifies an image) dynamics. Furthermore, most models struggle in semi-supervised settings where a portion of the data is unlabeled.

Overview of the multi-modal tasks addressed by HGAT, including Image Captioning and Visual Question Answering.

Methods

The authors propose HGAT, a two-stage architecture that utilizes Graph Neural Networks (GNNs) to represent relationships between different image-text samples within a task.

Stage 1: Modal-specific GNNs & Co-learning

The model constructs two separate graphs: one for visual features and one for semantic (text) features. A key innovation is the Attention-based Co-learning Framework. Rather than updating visual nodes based solely on visual similarity, the model employs attention patterns from the semantic graph to guide the visual graph, and vice versa. This allows the model to "borrow" context from one modality to resolve ambiguities in the other.

Stage 2: Relation-aware GNNs

The refined features from the first stage are concatenated and processed by a final GNN. This stage focuses on high-level relationships between samples to predict the correct label or answer for a new "query" sample.

Semi-supervised Extension

HGAT naturally handles unlabeled data. Unlabeled samples are initialized with zero-vector labels, allowing the graph attention mechanism to propagate information from labeled neighbors to unlabeled ones during the learning process.

Key Insights

  • Cross-Modal Synergy: Visual similarity alone is often insufficient. For instance, two images showing "a computer on a desk" may look similar, but their associated questions might differ entirely. By using semantic attention to weight visual features, the model identifies which visual details are relevant to the specific task.
  • Hierarchical Modeling: Separating the problem into modality-specific refinement followed by global relation modeling significantly outperforms monolithic fusion approaches.
  • Robustness: The hierarchical structure provides resilience against "distractor" samples—unlabeled samples from irrelevant classes—compared to standard GNN methods.

Results

HGAT was evaluated on Toronto COCO-QA, Visual Genome-QA, and COCO-FITB. It consistently outperformed baselines such as Prototypical Networks and standard GNNs.

Examples of the Image Captioning task where HGAT must relate visual regions to semantic descriptions.
  • VQA Performance: On the Visual Genome-QA 5-way 5-shot task, HGAT achieved 86.10% accuracy, surpassing the previous best baseline (DN4 at 84.25%).
  • Stability: HGAT demonstrated stable performance gains when scaling from 1-shot to 5-shot scenarios.
  • Semi-supervised Success: With only 40% of support data labeled, HGAT maintained a high accuracy of 67.16% on COCO-QA, significantly outperforming other graph-based models like EGNN.

Conclusion

While HGAT represents a significant advancement, future work could address its computational complexity, as fully-connected graphs can become memory-intensive with larger tasks. Additionally, the model currently relies on fixed pre-trained embeddings. Extending this hierarchical graph approach to other multi-modal domains, such as video-audio learning or medical imaging, remains a promising direction.

Summary updated

Abstract

Deep learning has made tremendous success in computer vision, natural language processing and even visual-semantic learning, which requires a huge amount of labeled training data. Nevertheless, the goal of human-level intelligence is to enable a model to quickly obtain an in-depth understanding given a small number of samples, especially with heterogeneity in the multi-modal scenarios such as visual question answering and image captioning. In this paper, we study the few-shot visual-semantic learning and present the Hierarchical Graph ATtention network (HGAT). This two-stage network models the intra- and inter-modal relationships with limited image-text samples. The main contributions of HGAT can be summarized as follows: 1) it sheds light on tackling few-shot multi-modal learning problems, which focuses primarily, but not exclusively on visual and semantic modalities, through better exploitation of the intra-relationship of each modality and an attention-based co-learning framework between modalities using a hierarchical graph-based architecture; 2) it achieves superior performance on both visual question answering and image captioning in the few-shot setting; 3) it can be easily extended to the semi-supervised setting where image-text samples are partially unlabeled. We show via extensive experiments that HGAT delivers state-of-the-art performance on three widely-used benchmarks of two visual-semantic learning tasks.
BibTeX
@inproceedings{yin2021hierarchical,
  author = {Chengxiang Yin and Kun Wu and Zhengping Che and Bo Jiang and Zhiyuan Xu and Jian Tang},
  title = {{Hierarchical Graph Attention Network for Few-shot Visual-Semantic Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00218},
}
Figure