ViGPTQA - State-of-the-Art LLMs for Vietnamese Question Answering: System Overview, Core Models Training, and Evaluations

Minh Thuan Nguyen, Khanh Tung Tran, Nhu Van Nguyen, Xuan-Son VuEMNLP 2023

TL;DR

This paper introduces ViGPTQA, a question-answering system for Vietnamese, and ViGPT, a novel instruction-tuned LLM specifically for Vietnamese. They address the scarcity of resources for Vietnamese NLP by creating new datasets and demonstrating state-of-the-art performance on various tasks, highlighting the need for language-specific LLMs.

What Problem Does This Paper Address?

Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP), but their development and application have largely focused on high-resource languages like English. Vietnamese, a low-resource language, suffers from a lack of training data and benchmark datasets, hindering the creation of effective LLM-powered applications. Existing multilingual LLMs often struggle with the unique linguistic nuances of Vietnamese, and prior evaluation efforts have not adequately addressed this gap. This paper aims to bridge this divide by developing and evaluating LLMs tailored for Vietnamese, specifically for question-answering systems.

Methods

The core of this work is the development of ViGPT, an instruction-tuned LLM for Vietnamese. The authors employed a multi-stage fine-tuning approach:

  • Initial Fine-tuning: They started by translating a large English instruction-following dataset (Alpaca) into Vietnamese using GPT-3.5-turbo. To address potential translation inaccuracies and capture Vietnamese-specific nuances, they augmented this with a dataset of native Vietnamese user-generated question-answering samples, as well as samples from the VinewsQA dataset and synthetic abstractive question-answering samples in the Vietnamese law domain. This combined dataset was used to fine-tune a base LLM, resulting in ViGPT-v2.
  • Domain Adaptation (ViGPT-Law): For specialized applications like legal question answering, they further adapted ViGPT. This involved pre-training on a large corpus of Vietnamese law documents and then fine-tuning with a mix of translated Alpaca data and synthetic law-specific question-answering pairs. This resulted in ViGPT-Law-v1 and ViGPT-Law-v2.
  • ViGPTQA System: The ViGPTQA system integrates the LLM with an embedding module and a vector database. This allows the system to retrieve relevant legal documents to provide factual and referenced answers to user queries.

Datasets

The paper introduces several new datasets and utilizes existing ones:

  • Vietnamese Alpaca Instruction-Following Data: A translated version of the English Alpaca dataset (52K samples).
  • Native Vietnamese QA Data: 1107 question-answering samples created by native Vietnamese users, covering history, geography, and literature.
  • VinewsQA Dataset: A subset of 3000 samples for extractive question answering.
  • ViLawsQA Training Set: 5000 synthetic abstractive question-answering samples generated for the Vietnamese law domain.
  • Vietnamese Law Documents Data: A large corpus of 252,425 Vietnamese law documents used for domain adaptation.
  • ViLawsQA Test Set: A benchmark of 1020 samples for evaluating abstractive question answering in the Vietnamese law domain.
  • ViTruthfulQA: A new dataset designed to evaluate the truthfulness of LLMs, focusing on Vietnamese knowledge and adversarial queries.
  • Vicuna-Instructions-80: A translated dataset of 80 challenging questions used for automatic evaluation with LLMs.

Key Insights and Ideas

  • Language-Specific LLMs are Crucial: The results strongly suggest that dedicated Vietnamese LLMs (like ViGPT) outperform general multilingual models on Vietnamese-specific tasks, highlighting the limitations of relying solely on cross-lingual transfer.
  • Data Augmentation is Key: Combining translated data with native user-generated and domain-specific synthetic data significantly improves LLM performance for low-resource languages.
  • Domain Adaptation Yields Significant Gains: Fine-tuning on domain-specific corpora (like Vietnamese law) dramatically enhances the model's accuracy and relevance within that domain.
  • Context Retrieval is Vital for QA: For accurate question answering, especially in specialized domains, integrating retrieval mechanisms (like vector databases) with LLMs is essential.
  • LLM-as-a-Judge is a Viable Evaluation Method: Using powerful LLMs like GPT-4 for automatic evaluation shows strong agreement with human judgments, offering a scalable way to assess model performance.

Key Results

  • Abstractive Question Answering (ViLawsQA): ViGPT-Law-v2, when provided with retrieved context, achieved a human evaluation score of 2.52 out of 4, outperforming other models and demonstrating the effectiveness of their domain adaptation and retrieval strategy.
  • Extractive Question Answering: ViGPT-v2 performed competitively with state-of-the-art models on various Vietnamese MRC datasets, often matching or exceeding them in F1 and human scores.
  • Named Entity Recognition (NER): Even with a limited few-shot scenario (100 samples), ViGPT-v1 achieved an F1-score of 69.31, demonstrating its practical utility with minimal training data.
  • Truthfulness (ViTruthfulQA): ViGPT-v2 achieved a human evaluation score of 25.45, showing promising truthfulness capabilities, though still lagging behind GPT-3.5-turbo.
  • Automatic Evaluation (Vicuna-Instructions-80): ViGPT-v1 and ViGPT-v2 outperformed other Vietnamese LLMs, demonstrating their effectiveness in handling challenging, diverse questions.

Limitations and Future Work

  • Data Quality and Quantity: The authors acknowledge that the quality and quantity of instruction-following data for Vietnamese are still relatively low, limiting the model's ability to handle multiple tasks effectively within a single model.
  • Fairness and Bias: While ViGPTs show good truthfulness, there's still a gap compared to absolute truthfulness, and further evaluation is needed to address potential biases.
  • Performance Gap with Large Multilingual Models: ViGPT models still face a performance gap compared to massive models like GPT-3.5-turbo, primarily due to scale differences.
  • Future Work: The authors suggest scaling up ViGPT models to larger parameter counts and exploring more advanced techniques for data augmentation and bias mitigation. They also emphasize the potential for ViGPTs in various real-world applications beyond question answering.

Summary updated

BibTeX
@inproceedings{nguyen2023vigptqa,
  author = {Minh Thuan Nguyen and Khanh Tung Tran and Nhu Van Nguyen and Xuan-Son Vu},
  title = {{ViGPTQA - State-of-the-Art LLMs for Vietnamese Question Answering: System Overview, Core Models Training, and Evaluations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure