TL;DR
This paper investigates how to build better multilingual question-answering systems that can understand queries in various languages and retrieve relevant information from documents. They found that using language-agnostic models like LaBSE for retrieval outperforms language-aware models, but surprisingly, using ChatGPT to rewrite queries didn't improve performance, likely due to topic switching and irrelevant information being included.
What Problem Does This Paper Address?
The internet is predominantly in English, creating a barrier for non-English speakers wanting to access information or use virtual assistants. Multilingual document-grounded question-answering (QA) systems aim to bridge this gap, allowing users to ask questions in their native language and get answers from documents. While English-centric systems are well-studied, extending them to multiple languages, especially with limited data, presents challenges. Specifically, this paper tackles two main issues: 1) effectively retrieving relevant documents across different languages for a given query, and 2) improving retrieval by rewriting user queries using conversational context, potentially with the help of large language models like ChatGPT.
Methods
The core of the paper focuses on the retriever component of a document-grounded QA system. They compare two paradigms for multilingual dense passage retrieval (mDPR) using bi-encoders:
- Language-Agnostic Paradigm: Uses models like LaBSE (Language-Agnostic BERT Sentence Embedding), which are trained to map sentences from different languages into a shared high-dimensional space. This allows for direct comparison of embeddings regardless of the original language.
- Language-Aware Paradigm: Uses models like XLM-R (Cross-lingual Language Model - RoBERTa), which are pre-trained on a large multilingual corpus but might maintain language-specific representations.
They pre-trained these models on English and Chinese data and then fine-tuned them on French and Vietnamese datasets.
Additionally, they explored query rewriting using ChatGPT. They prompted ChatGPT to rewrite the current user query by incorporating the preceding dialogue history, aiming to create a more informative standalone question.
Datasets
The study utilizes the DialDoc 2023 shared task dataset, which includes:
- 797 Vietnamese dialogues (average 4 turns per dialogue).
- 816 French dialogues (average 5 turns per dialogue). These dialogues are grounded in documents from nine different domains (e.g., Technology, Health, Insurance). Each dialogue turn includes role annotations, and the data is provided in reverse chronological order. The dataset includes queries, dialogue context, positive passages (containing the answer), and negative passages (related but not containing the answer).
Key Insights and Ideas
- Language-agnostic retrieval reigns supreme: The paper strongly advocates for language-agnostic models like LaBSE for multilingual retrieval. Their ability to create a unified embedding space seems crucial for effective cross-lingual transfer.
- ChatGPT's query rewriting falls short (for now): Despite the hype around LLMs, ChatGPT's query rewriting did not improve retrieval performance on the tested samples. This is a significant finding, suggesting that simply feeding conversational history to an LLM for rewriting isn't a silver bullet.
- Topic switching is a major hurdle: The authors pinpoint topic switching in the later turns of a dialogue as a primary reason for the failure of query rewriting. When the conversation shifts, rewritten queries might incorporate irrelevant context, confusing the retriever.
- Summarization vs. Specificity: ChatGPT's tendency to summarize both relevant and irrelevant parts of the conversation, and even "hallucinate" information, leads to less precise and effective rewritten queries.
Key Results
- Retrieval Performance: The LaBSE-based mDPR retriever consistently outperformed the XLM-R-based mDPR retriever across various fine-tuning configurations (French, Vietnamese, and combined). For instance, in one configuration (Table 1), LaBSE achieved R@1 scores of 0.75, while XLM-R achieved 0.65.
- Query Rewriting Impact: When evaluating query rewriting (Table 2), using raw queries (original dialogue history + current turn) led to significantly better performance than queries rewritten by ChatGPT. For example, when evaluated on French, raw queries with LaBSE achieved an R@1 of 0.57, whereas ChatGPT-rewritten queries achieved only 0.16.
- Reader Performance (Appendix): The appendix shows that using Fusion-in-Decoder (FiD) with the mT5 reader improved performance metrics (F1, BLEU, Rouge-L) compared to a vanilla mT5 reader, demonstrating the benefit of more sophisticated reader architectures.
Limitations and Future Work
- Limited Sample Size: The query rewriting experiments were conducted on a sample of 2000 queries, which the authors acknowledge limits the generalizability of their findings.
- Context Window and Prompting: Constraints on ChatGPT's context window and the manual nature of prompt creation (leading to potential human bias) prevented testing more advanced prompting techniques like in-context examples.
- Future Directions:
- Investigate advanced prompting methods to improve query rewriting quality.
- Explore open-source models and advanced fine-tuning techniques (e.g., Prefix Tuning, Prompt Tuning) for query rewriting.
- Further research into the "hallucination" issue with LLMs and its impact on query rewriting accuracy.
- Analyze potential societal biases within the dataset to improve reliability.
Summary updated