TL;DR
This paper introduces RAC, a novel dataset and framework for Retrieval-Augmented Conversation (RAC), a more realistic form of conversational question answering that requires retrieving external knowledge at each turn. They also present a strong baseline system and conduct extensive analyses to understand the interplay between retrieval and response generation in conversational settings.
What Problem Does This Paper Address?
Conversational Question Answering (CQA) systems aim to answer questions within a dialogue. However, existing CQA datasets often rely on predefined contexts, limiting their realism. The authors propose Retrieval-Augmented Conversation (RAC) as a more practical scenario where systems must retrieve relevant information from external knowledge sources at each turn to answer questions. This involves two key challenges: effectively reformulating queries based on conversational history for better retrieval, and then using the retrieved knowledge to generate accurate, human-like responses. While conversational search and retrieval-augmented generation (RAG) have been studied independently, a comprehensive approach to RAC has been underexplored.
Methods
The paper introduces a pipeline-based system for RAC, comprising four stages:
- Query Rewriting: A generative model is trained to rewrite user queries, incorporating conversational history. This model is trained in multiple stages, first on human-written queries and then on pseudo-queries generated for a larger passage collection, effectively augmenting the training data.
- Retrieval: BM25 is used as the primary retriever due to its effectiveness with term-based queries, which are expected from the rewritten queries. A dense retriever is also explored.
- Reranking: Instead of training a separate cross-encoder for reranking, the authors leverage the trained query rewriting model. The intuition is that a model trained to generate queries from passages will assign higher probabilities to passages that are genuinely relevant to a query.
- Response Generation: The Fusion-in-Decoder (FiD) architecture is employed, which independently encodes retrieved passages and then uses a decoder to generate responses. They also experiment with a large language model (LLM) for response generation.
Datasets
The authors introduce a new dataset called RAC, derived from a Korean knowledge-retrieval conversation dataset available on AI-Hub. This dataset includes conversations with supporting factors and referred document URLs. To address the limitations of the original dataset for retrieval, they:
- Crawled Korean Wikipedia and publicly available news data to create a passage collection of 1,345,209 passages.
- Collected 10,266 human-written queries, specifically designed to be suitable for retrieval and to avoid including key terms from the answers.
- Annotated additional relevant passages, resulting in 17,606 such passages.
The preprocessed dataset consists of 1,203 conversations, with an average of 3.47 relevant passages per turn.
Key Insights and Ideas
- Bridging the CQA-RAC Gap: The paper highlights the need for a more realistic CQA setting that incorporates external knowledge retrieval at each turn, which they term RAC.
- Leveraging Query Rewriting for Reranking: A novel approach of using the query rewriting model itself for reranking passages is proposed and shown to be effective, avoiding the cost of training separate cross-encoders.
- BM25's Advantage with Human-Written Queries: BM25 outperforms the dense retriever when using human-written queries, likely because these queries are concise and term-focused, aligning well with BM25's strengths.
- Retrieval Quality is Crucial for Generation: While LLMs show promise, the quality of generated responses is still heavily dependent on the quality of retrieved passages, even for advanced models.
- Temporal Questions are Challenging: Human evaluation reveals that current models struggle with temporal questions, often producing irrelevant or partially relevant responses.
Key Results
- The proposed reranking strategy significantly improved first-stage retrieval results for both dense and BM25 retrievers, with over half of queries retrieving relevant passages within the top five.
- The BM25 retriever generally outperformed the dense retriever in retrieval metrics.
- Response generation performance improved with better retrieval results, and responses generated using passages retrieved by the dense retriever scored higher than those from BM25, despite BM25's better retrieval performance.
- Training the query rewriting model on passages led to a performance improvement of 0.022%p in MRR compared to training without passages.
- Human evaluation indicated that while models can generate relevant responses, irrelevance was a significant issue, particularly for temporal questions.
Limitations and Future Work
- Model Size: The query rewriting model used is a small encoder-decoder, and the authors suggest that employing larger LLMs could lead to more effective queries.
- Language Specificity: The dataset is in Korean, and experiments on diverse languages are needed for language-agnostic generalization. The authors plan to translate the dataset into English.
- End-to-End Systems: The current pipeline approach could be improved by integrating retriever and response generation models into an end-to-end system for potentially better performance.
- Temporal Question Handling: The models' weakness in handling temporal questions necessitates more sophisticated strategies.
- Evaluation Metrics: Existing evaluation metrics may not fully capture the nuances of diverse responses, suggesting a need for better evaluation methods.
Summary updated