TL;DR
This paper introduces OMG-QA, a challenging new benchmark for open-domain, multi-modal question answering that requires systems to retrieve and reason across text, tables, and images. They also present a comprehensive evaluation of various Retrieval Augmented Generation (RAG) systems, revealing significant limitations in current approaches for handling complex, multi-modal information needs.
What Problem Does This Paper Address?
Modern question answering (QA) systems typically fall into two categories: closed-domain QA, which operates on a given context, and open-domain QA, which retrieves information from large knowledge sources. While open-domain QA has advanced significantly, especially with the advent of Large Language Models (LLMs), a critical gap remains in evaluating systems that need to handle multi-modal information (text, tables, images) and reason across multiple documents to generate detailed, long-form answers. Existing benchmarks often focus on single modalities or factoid questions, failing to capture the complexity of real-world information-seeking tasks that require synthesizing information from diverse sources and formats. This paper aims to bridge this gap by introducing OMG-QA, a dataset designed to push the boundaries of RAG systems in these challenging multi-modal, multi-document scenarios.
Methods
The authors propose a two-stage approach for building their OMG-QA dataset and evaluating QA systems:
- Question Generation Pipelines:
- Pipeline 1 (Text and Table): This pipeline leverages Wikipedia articles with substantial text and multiple tables. It uses OpenAI's text-embedding-ada-002 to identify relevant article pairs and then employs GPT-4 to generate questions that require integrating information from both text and tables.
- Pipeline 2 (Text, Tables, and Images): This pipeline focuses on incorporating images as well. It starts by extracting a document's table of contents and visually representing its structure. Then, it identifies tables and images within sections and uses GPT-4 to generate questions that necessitate retrieving information from at least two different modalities within a single document.
- Document and Evidence Retrieval Annotation:
- To create a comprehensive pool of relevant evidence, they deploy various QA systems to retrieve documents and evidence for each query.
- These retrieved items are then deduplicated and pooled.
- Human annotators (and later, GPT-4) evaluate the relevance of each piece of evidence to the corresponding query, establishing a ground truth for evaluation.
- System Evaluation:
- The paper evaluates a diverse set of QA systems by systematically varying components like document retrievers (Wikipedia API vs. DuckDuckGo API), evidence indexers (text-only, textual-visual, modality-specific), evidence retrievers (sparse, dense, generative, hybrid), and LLMs.
- They also compare one-round (standard RAG) vs. multi-round (LLM Agent) retrieval strategies.
- Evaluation metrics include evidence relevancy, correct usage of evidence, and citation completeness.
Datasets
The primary contribution in terms of datasets is the introduction of OMG-QA, an open-domain, multi-modal generative question answering dataset.
- OMG-QA: Contains 1,000 questions, with 500 generated from each of the two pipelines described above. Each question is linked to an average of 10 relevant documents and 33 pieces of relevant evidence. The dataset specifically targets scenarios requiring the integration of text, tables, and images.
Key Insights and Ideas
- Multi-modal is Hard: The core insight is that current RAG systems struggle significantly when faced with the need to integrate information from multiple modalities (text, tables, images) and across multiple documents. OMG-QA highlights this challenge effectively.
- Multi-index Wins: For this multi-modal dataset, systems using multiple indices (e.g., separate indices for text and images) consistently outperform single-index setups. This is because a single index struggles to efficiently represent and retrieve diverse modalities.
- LLM Choice Matters Less Than Indexing/Retrieval: Surprisingly, the choice of LLM (even between smaller and much larger models) had a minimal impact on retrieval performance. This suggests that the architecture of the indexing and retrieval components plays a more crucial role.
- Multi-round Retrieval's Double-Edged Sword: While multi-round retrieval (LLM agents) improves document recall, it can sometimes degrade evidence retrieval performance. The hypothesis is that the constant re-ranking in subsequent rounds might inadvertently discard relevant evidence.
- GPT-4 as an Evaluator: The paper demonstrates a high agreement between human judgments and GPT-4 evaluations for tasks like evidence relevancy, correct evidence usage, and citation completeness, suggesting GPT-4's potential as a scalable evaluator.
Key Results
- Document Retrieval: DuckDuckGo's search API consistently outperformed Wikipedia's own API for document retrieval across various LLM configurations.
- Evidence Retrieval: Multi-index setups significantly improved evidence retrieval performance compared to text-only indices. Dense retrievers (especially those using models like
snowflake-arctic-embed-1andgte-large-en-v1.5) generally outperformed sparse and generative retrievers. - LLM Impact: The choice of LLM had a negligible impact on retrieval performance, with models like Llama-3-8b performing similarly to larger models.
- Retrieval Strategy: Multi-round retrieval improved document recall but sometimes led to a decrease in evidence retrieval performance.
- Best Performing System: The top-performing system combined DuckDuckGo for document retrieval, modality-specific indexing,
gte-large-en-v1.5for text/table, CLIP for images, and GPT-4 for LLM tasks, utilizing multi-round retrieval with memory and self-reflection. - Answer/Citation Quality: Proprietary models like GPT-3.5-Turbo and GPT-4 excelled in effective retrieval usage and citation completeness. GritLM-8x7b showed strong performance in the relevance and accuracy of attributed evidence.
Limitations and Future Work
- Single Modality Solvability: Some questions in OMG-QA might be solvable using only a single modality, which could challenge the necessity of a strictly multi-modal approach for all questions.
- GPT-4 Evaluation Staticness: GPT-4 evaluates evidence in isolation, lacking the dynamic, iterative adjustment that human annotators can perform when assessing relevance in the context of a multi-hop question. This static approach might lead to overlooking relevant evidence.
- Future Work: The authors position OMG-QA as a benchmark to drive the development of more capable QA systems that can better handle open-domain, multi-modal information-seeking tasks. Future work would likely involve developing more robust multi-modal RAG architectures and exploring more effective multi-round retrieval strategies.
Summary updated