Abstract
The Hierarchical Graph Attention Network (HGAT) is a novel two-stage framework designed to solve multi-modal tasks like Visual Question Answering (VQA) using only a handful of examples. By explicitly modeling both the relationships within a single modality and the "co-learning" cross-talk between images and text, HGAT achieves state-of-the-art performance in few-shot and semi-supervised settings.
Introduction
Deep learning typically requires massive datasets to function effectively. However, humans possess the remarkable ability to look at a single photo and a related snippet of text to immediately understand the context—a capability known as few-shot visual-semantic learning. This remains a significant hurdle for artificial intelligence.
Existing approaches often attempt to adapt general meta-learning algorithms, such as MAML, to multi-modal problems. However, these methods frequently treat visual and text data as flat features, failing to capture the intricate "intra-modal" (relationships within images) and "inter-modal" (how a caption clarifies an image) dynamics. Furthermore, most models struggle in semi-supervised settings where a portion of the data is unlabeled.
Methods
The authors propose HGAT, a two-stage architecture that utilizes Graph Neural Networks (GNNs) to represent relationships between different image-text samples within a task.
Stage 1: Modal-specific GNNs & Co-learning
The model constructs two separate graphs: one for visual features and one for semantic (text) features. A key innovation is the Attention-based Co-learning Framework. Rather than updating visual nodes based solely on visual similarity, the model employs attention patterns from the semantic graph to guide the visual graph, and vice versa. This allows the model to "borrow" context from one modality to resolve ambiguities in the other.
Stage 2: Relation-aware GNNs
The refined features from the first stage are concatenated and processed by a final GNN. This stage focuses on high-level relationships between samples to predict the correct label or answer for a new "query" sample.
Semi-supervised Extension
HGAT naturally handles unlabeled data. Unlabeled samples are initialized with zero-vector labels, allowing the graph attention mechanism to propagate information from labeled neighbors to unlabeled ones during the learning process.
Key Insights
- Cross-Modal Synergy: Visual similarity alone is often insufficient. For instance, two images showing "a computer on a desk" may look similar, but their associated questions might differ entirely. By using semantic attention to weight visual features, the model identifies which visual details are relevant to the specific task.
- Hierarchical Modeling: Separating the problem into modality-specific refinement followed by global relation modeling significantly outperforms monolithic fusion approaches.
- Robustness: The hierarchical structure provides resilience against "distractor" samples—unlabeled samples from irrelevant classes—compared to standard GNN methods.
Results
HGAT was evaluated on Toronto COCO-QA, Visual Genome-QA, and COCO-FITB. It consistently outperformed baselines such as Prototypical Networks and standard GNNs.
- VQA Performance: On the Visual Genome-QA 5-way 5-shot task, HGAT achieved 86.10% accuracy, surpassing the previous best baseline (DN4 at 84.25%).
- Stability: HGAT demonstrated stable performance gains when scaling from 1-shot to 5-shot scenarios.
- Semi-supervised Success: With only 40% of support data labeled, HGAT maintained a high accuracy of 67.16% on COCO-QA, significantly outperforming other graph-based models like EGNN.
Conclusion
While HGAT represents a significant advancement, future work could address its computational complexity, as fully-connected graphs can become memory-intensive with larger tasks. Additionally, the model currently relies on fixed pre-trained embeddings. Extending this hierarchical graph approach to other multi-modal domains, such as video-audio learning or medical imaging, remains a promising direction.
Summary updated