X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment

Dongjae Shin, HyeonSeok Lim, Inho Won, ChangSu Choi, Minjun Kim, Seungwoo Song, Hangyeol Yoo, Sangmin Kim, Kyungtae LimNAACL 2024 · DOI

TL;DR

This paper introduces X-LLaVA, a novel bilingual large vision-language model that significantly improves performance in Korean and English. It achieves this through a cost-effective data generation pipeline using GPT-4V and a refined training framework that includes vocabulary expansion and multilingual pretraining.

What Problem Does This Paper Address?

The rapid advancement of large multimodal models (LMMs) has opened up exciting possibilities, but their development is hampered by the high cost and complexity of creating training data, especially for multiple languages. Existing LMMs often struggle with multilingual capabilities, primarily focusing on English. While some models attempt multilingual support, they often lack sufficient visual information or are limited to specific languages. This paper addresses the challenge of building efficient, cost-effective, and high-performing multilingual LMMs, particularly focusing on the alignment between Korean and English.

Methods

The authors propose a two-pronged approach:

  1. Cost-Effective Multilingual VIF Dataset Construction: They leverage GPT-4V to automatically generate a diverse multilingual visual instruction-following (VIF) dataset. This dataset, named mvif, focuses on object relational information and includes four types of data: object-centric, location-centric, atmosphere-centric, and conversational. This method is designed to be more efficient and capture richer visual details than previous approaches.
  2. X-LLaVA Model Architecture and Training: X-LLaVA builds upon the LLaVA1.5 architecture and incorporates three key enhancements:
  3. Vocabulary Expansion: To address the limited Korean vocabulary in existing models, they expand the LLaMA2 tokenizer by adding 7,478 words from the KoBERT vocabulary.
  4. Bilingual Pretraining: They conduct additional pretraining using both English and Korean Wikipedia data to foster knowledge association between the languages.
  5. Multilingual VIF Training: The model is trained on the newly constructed mvif dataset, along with translated versions of existing VIF datasets, to enhance its multilingual visual instruction-following capabilities.

Datasets

  • Training Datasets:
    • mvif: A newly constructed 91K English-Korean-Chinese multilingual VIF dataset.
  • LLaVA-CC3M (English) and its Korean translation (LLaVA-KOCC3M).
  • LLaVA instruct dataset (English) and its Korean translation.
  • English Wikipedia data (Wen) and Korean Wikipedia data (Wko) for pretraining.
  • Evaluation Datasets:
    • English: VQA2.0, GQA, LV (LLaVAW), POPE.
  • Korean: KoViz, KoLiv.
  • Bilingual: BVQA (English-Korean).

Key Insights and Ideas

  • GPT-4V as a Cost-Effective Data Generator: Using GPT-4V for data generation, especially with direct image input, is shown to be a powerful and efficient method for creating rich, multilingual VIF datasets.
  • Vocabulary Expansion is Crucial for Multilingual Performance: Simply using existing models with limited vocabularies for specific languages (like Korean) leads to issues like code-switching. Expanding the vocabulary significantly boosts performance.
  • Bilingual Pretraining Bridges Knowledge Gaps: Explicitly training on both English and Korean data helps the model learn cross-lingual associations, leading to improved performance in both languages.
  • Multilingual VIF Enhances Nuance: The mvif dataset, with its focus on object relations and diverse question types, helps the model understand images more deeply and generate more nuanced responses.
  • Human vs. GPT Evaluation Nuances: While GPT evaluations are useful, they can be more conservative and prone to "ties" compared to human evaluations, especially when dealing with ambiguous cases or cultural nuances.

Key Results

  • Quantitative Improvements: X-LLaVA demonstrated an average improvement of approximately 5.2% in three Korean quantitative evaluations compared to previous models. It also achieved the highest performance in two out of five English quantitative evaluations.
  • Qualitative Superiority: Preference assessments using GPT-4V showed that X-LLaVA generated responses in both English and Korean that were 19-93% superior to existing models.
  • Efficiency: The entire training process for X-LLaVA, including data generation and model training, was completed in approximately 7.5 days using a single A6000 GPU, with the 91K training data costing around $3,200.
  • Performance Comparison: X-LLaVA outperformed BLIP-2, InstructBLIP, LLaVA1.5, and KoLLaVA in Korean evaluations, scoring an average of 57.0 points. It also showed strong performance in English evaluations, particularly on GQA and BVQA.

Limitations and Future Work

  • Chinese Component: The Chinese component of the model did not undergo vocabulary expansion, making it more akin to a Korean-English enhanced model. Further research is needed for models with vocabulary enhancement for more than three languages.
  • Chinese VQA Data: The lack of publicly available Chinese VQA evaluation data hindered diverse assessments for the Chinese language.
  • Future Directions: The authors suggest further investigation into models that have undergone vocabulary enhancement and knowledge connection for more than three languages. They also highlight the need for more comprehensive Chinese VQA datasets.

Summary updated

BibTeX
@inproceedings{shin2024x,
  author = {Dongjae Shin and HyeonSeok Lim and Inho Won and ChangSu Choi and Minjun Kim and Seungwoo Song and Hangyeol Yoo and Sangmin Kim and Kyungtae Lim},
  title = {{X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.158},
}
Figure