Abstract
Visual Question Answering (VQA) typically relies on fusing high-level features from the final layers of deep networks. However, this approach neglects the rich, multi-level abstractions captured in intermediate layers. We present the Quaternion Block Network (QBN), which utilizes Hamilton quaternion products to model interactions across all intermediate layers. By grouping layers into quaternions, QBN captures complex cross-layer relationships efficiently. Our results on the VQA v2.0 benchmark demonstrate that QBN outperforms larger pre-trained models like ViLBERT, achieving a test-standard accuracy of 71.02%.
Introduction: The Gap in Multi-Layer Interaction
In the field of Visual Question Answering (VQA), models must bridge the gap between computer vision and natural language processing. Current state-of-the-art architectures, such as MCAN or DFAF, employ stacked attention layers to process these modalities. A significant limitation of these models is their reliance on features from only the final layer for answer prediction.
Intermediate layers capture varying levels of semantic abstraction. By ignoring these "middle" interactions, models fail to achieve a holistic understanding of the data. The challenge lies in modeling these complex, cross-layer interactions without incurring prohibitive computational costs.
Methodology: The Quaternion Block Network (QBN)
The core innovation of QBN is the treatment of neural network layers as components of a quaternion—a four-dimensional number system ($r + i\mathbf{i} + j\mathbf{j} + k\mathbf{k}$).
Quaternion Representation and the Hamilton Product
QBN groups four consecutive layers of features into a single quaternion. For instance, four layers of text features form a "text quaternion," and four layers of visual features form a "visual quaternion." Instead of standard matrix multiplication, the model employs the Hamilton product. This mathematical operation forces every layer in one modality to interact with every layer in the other, capturing multi-layer relationships in a compact representation.
System Architecture
The overall architecture integrates several key components to facilitate deep multimodal fusion.
- Multi-layer Content Learning: An RNN extracts a holistic summary of text features across all layers, providing global context to update visual information via co-attention.
- Multi-layer Relationship Learning: This module uses the quaternion product to model the inter-dependencies between layers.
- Dynamic Spatial Scaling: Visual features are scaled and biased based on the question vector, focusing the model on relevant image regions early in the process.
Experimental Setup and Results
The model was evaluated on the VQA v2.0 benchmark, utilizing Visual Genome (VG) for additional training and GloVe for initial word embeddings.
Performance
QBN achieved a state-of-the-art test-standard accuracy of 71.02%, surpassing both MCAN (70.90%) and the large-scale pre-trained ViLBERT (70.92%).
Ablation and Qualitative Analysis
Ablation studies confirmed that both Relationship Learning (quaternion-based) and Content Learning (holistic text summary) are essential. Removing either component resulted in a performance drop of approximately 0.15% to 0.20%. Qualitatively, attention maps generated by QBN showed progressive refinement through the quaternion blocks, leading to more accurate object localization compared to baseline models.
Discussion and Future Work
While effective, the QBN architecture is currently constrained by a fixed block size of four, dictated by the dimensions of a quaternion. Future research could explore more flexible grouping strategies. Additionally, while theoretically efficient, quaternion operations benefit significantly from specialized software implementations.
The authors suggest that multi-layer quaternion interaction could be extended to other multimodal tasks, such as visual grounding and general vision-language pre-training, where deep cross-modal understanding is paramount.
Summary updated