Semantic Aligned Multi-modal Transformer for Vision-LanguageUnderstanding: A Preliminary Study on Visual QA

Han Ding, Li Erran Li, Zhiting Hu, Yi Xu, Dilek Hakkani-Tür, Zheng Du, Belinda Zeng2021 · DOI

Abstract

Recent vision-language understanding approaches adopt a multi-modal transformer pretraining and finetuning paradigm. Prior work learns representations of text tokens and visual features with cross-attention mechanisms and captures the alignment solely based on indirect signals. In this work, we propose to enhance the alignment mechanism by incorporating image scene graph structures as the bridge between the two modalities, and learning with new contrastive objectives. In our preliminary study on the challenging compositional visual question answering task, we show the proposed approach achieves improved results, demonstrating potentials to enhance vision-language understanding.

Introduction

Current state-of-the-art vision-language models, such as ViLBERT or UNITER, excel at identifying objects within an image but often struggle to comprehend the complex relationships between them. These models typically treat images as collections of regions and text as sequences of words, relying on attention mechanisms to infer mappings. However, this mapping is frequently learned from indirect signals, leading to semantic ambiguity. For instance, a model might correctly identify a "cup," "tray," and "table" but fail to determine that the cup is situated on the table rather than the tray. To achieve true scene understanding, models require a structured method to align visual relationships with linguistic descriptions.

Methodology: The Samformer Architecture

The authors introduce Samformer (Semantic Aligned Multi-modal Transformer), which integrates Visual Scene Graphs into the transformer architecture to act as a structural bridge between modalities.

The Scene Graph Bridge

Using an off-the-shelf scene graph generator, Samformer extracts triplets—(subject, relation, object)—from images. These triplets are naturally "grounded," meaning the word "woman" is linked to a specific bounding box. This provides an explicit roadmap for the transformer's attention, aligning raw pixels with natural language.

Figure 5: Samformer architecture incorporating scene graph triplets and triplet alignment.

Pre-training Objectives

To enforce the use of this bridge, the model utilizes three primary training tasks:

  1. Masked Language Modeling (MLM): The model predicts missing words in a sentence by synthesizing information from both the image and the scene graph triplets.
  2. Cross-Modal Contrastive Loss (Global - CMCG): The model must distinguish between a scene graph that accurately matches an image and a "fake" graph sampled from a different image.
  3. Cross-Modal Contrastive Loss (Triplet-level - CMCT): The model evaluates individual triplets (e.g., "man-riding-bike") to identify if they are original or have been corrupted with incorrect labels.

Experimental Setup and Datasets

Samformer was pre-trained on a massive corpus including Microsoft COCO Captions, Visual Genome, VQA, GQA, Flickr30k, SBU, and Conceptual Captions. Evaluation focused on the GQA (Geometric Question Answering) dataset, which tests compositional reasoning through questions like "Is the red bowl to the left of the blue plate?"

Key Results and Analysis

Samformer demonstrated significant improvements over the strong baseline model, Oscar:

  • Overall Accuracy: Samformer improved GQA accuracy by 2.11% on the Test-dev set and 1.61% on the Test-std set.
  • Open Questions: The model showed its greatest strength in non-binary "open" questions, achieving a 4.09% improvement. These questions require identifying specific attributes or relationships, where the scene graph bridge is most effective.
  • The "Oracle" Potential: As shown in Figure 6, providing the model with "ground-truth" (perfect) scene graphs leads to a dramatic spike in performance. This indicates that as scene graph generation technology matures, the Samformer framework will become increasingly powerful.
Figure 6: Evaluation accuracy on GQA comparing models with predicted, ground truth, and no relation tags.

Conclusion and Future Work

The primary limitation of the current approach is its dependence on the quality of the external scene graph generator. The model's performance is currently capped by the accuracy and vocabulary of these off-the-shelf tools.

Future research will explore joint training, where the scene graph generator and the multi-modal transformer are optimized simultaneously. This co-optimization would allow the model to learn visual features specifically tailored for the complex relationships it needs to identify, further closing the gap between visual perception and linguistic understanding.

Summary updated

Abstract

Recent vision-language understanding approaches adopt a multi-modal transformer pretraining and finetuning paradigm. Prior work learns representations of text tokens and visual features with cross-attention mechanisms and captures the alignment solely based on indirect signals. In this work, we propose to enhance the alignment mechanism by incorporating image scene graph structures as the bridge between the two modalities, and learning with new contrastive objectives. In our preliminary study on the challenging compositional visual question answering task, we show the proposed approach achieves improved results, demonstrating potentials to enhance visionlanguage understanding.
BibTeX
@misc{ding2021semantic,
  author = {Han Ding and Li Erran Li and Zhiting Hu and Yi Xu and Dilek Hakkani-Tür and Zheng Du and Belinda Zeng},
  title = {{Semantic Aligned Multi-modal Transformer for Vision-LanguageUnderstanding: A Preliminary Study on Visual QA}},
  year = {2021},
  doi = {10.18653/v1/2021.maiworkshop-1.11},
}
Figure