Improving the Cross-Lingual Generalisation in Visual Question Answering

Farhad Nooralahzadeh, Rico SennrichAAAI 2023 · DOI

Introduction

Multimodal AI models have demonstrated impressive capabilities in Visual Question Answering (VQA), yet they frequently exhibit a pronounced "English bias." Because training datasets are predominantly in English, these models often fail when tasked with zero-shot cross-lingual VQA—answering questions about an image in languages like Bengali or Indonesian without specific training in those languages. This performance gap stems from three primary issues: binary loss functions that treat all errors equally, language-specific overfitting, and misaligned internal representations across different languages.

Methodology: The CLG-VQA Strategy

To address these challenges, researchers proposed CLG-VQA, a fine-tuning framework comprising three synergistic techniques:

  1. Linguistic Prior Objective: Standard cross-entropy loss penalizes a model equally for any incorrect answer. CLG-VQA introduces a similarity-based loss using semantic knowledge from WordNet or GloVe. This ensures the model is penalized less if its prediction is semantically related to the ground truth (e.g., predicting "skater" instead of "skateboarder").
  2. Task-Specific Sparse Fine-Tuning (SFT): Utilizing the "Lottery Ticket Hypothesis," the authors identify a sparse subnetwork (roughly 12-13% of parameters) optimized for VQA. Pruning non-essential weights helps the model ignore English-specific noise and focus on task-essential features.
  3. Multilingual Code-Mixing (CDM): This technique creates synthetic training data by replacing words in English questions with translations from target languages, forcing the model to align semantic concepts across different linguistic inputs.
Overview of the CLG-VQA framework, illustrating the integration of prior knowledge into the loss function and the model's predictive distribution.

Experimental Evaluation

The strategy was evaluated using xGQA, a multilingual benchmark covering seven diverse languages: Bengali (Bn), German (De), Indonesian (Id), Korean (Ko), Portuguese (Pt), Russian (Ru), and simplified Chinese (Zh). The researchers applied their methods to two state-of-the-art multilingual multimodal transformers, UC2 and M3P.

Key Results and Analysis

The proposed CLG-VQA strategy yielded significant improvements in zero-shot accuracy across all tested languages. For the UC2 model, the combination of Linguistic Priors, SFT, and CDM resulted in a +13.12% average accuracy gain. Similarly, the M3P model saw a +12.63% improvement.

Performance comparison of various model configurations across seven languages, highlighting the substantial gains achieved by the full CLG-VQA strategy (SFT+CDM+Prior).

Beyond raw accuracy, the study revealed that many "errors" were semantically plausible synonyms. When evaluation metrics were relaxed to accept synonyms, performance increased by an additional 1-4%, suggesting that models possess more cross-lingual understanding than standard metrics capture.

Conclusion

The success of CLG-VQA demonstrates that cross-lingual generalization can be significantly improved by moving beyond simple binary classification. By rewarding "less wrong" answers, focusing on task-specific subnetworks, and using synthetic code-mixing, the researchers have narrowed the performance gap between English and other languages in multimodal tasks. Future work will likely explore more flexible pruning rates and evaluation metrics that better reflect linguistic nuances.

Summary updated

BibTeX
@inproceedings{nooralahzadeh2023improving,
  author = {Farhad Nooralahzadeh and Rico Sennrich},
  title = {{Improving the Cross-Lingual Generalisation in Visual Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26574},
}
Figure