OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems

Linyong Nan, Weining Fang, Aylin Rasteh, Pouya Lahabi, Weijin Zou, Yilun Zhao, Arman CohanEMNLP 2024 · OpenReview

TL;DR

This paper introduces OMG-QA, a challenging new benchmark for open-domain, multi-modal question answering that requires systems to retrieve and reason across text, tables, and images. They also present a comprehensive evaluation of various Retrieval Augmented Generation (RAG) systems, revealing significant limitations in current approaches for handling complex, multi-modal information needs.

What Problem Does This Paper Address?

Modern question answering (QA) systems typically fall into two categories: closed-domain QA, which operates on a given context, and open-domain QA, which retrieves information from large knowledge sources. While open-domain QA has advanced significantly, especially with the advent of Large Language Models (LLMs), a critical gap remains in evaluating systems that need to handle multi-modal information (text, tables, images) and reason across multiple documents to generate detailed, long-form answers. Existing benchmarks often focus on single modalities or factoid questions, failing to capture the complexity of real-world information-seeking tasks that require synthesizing information from diverse sources and formats. This paper aims to bridge this gap by introducing OMG-QA, a dataset designed to push the boundaries of RAG systems in these challenging multi-modal, multi-document scenarios.

Methods

The authors propose a two-stage approach for building their OMG-QA dataset and evaluating QA systems:

  1. Question Generation Pipelines:
    • Pipeline 1 (Text and Table): This pipeline leverages Wikipedia articles with substantial text and multiple tables. It uses OpenAI's text-embedding-ada-002 to identify relevant article pairs and then employs GPT-4 to generate questions that require integrating information from both text and tables.
  2. Pipeline 2 (Text, Tables, and Images): This pipeline focuses on incorporating images as well. It starts by extracting a document's table of contents and visually representing its structure. Then, it identifies tables and images within sections and uses GPT-4 to generate questions that necessitate retrieving information from at least two different modalities within a single document.
  3. Document and Evidence Retrieval Annotation:
    • To create a comprehensive pool of relevant evidence, they deploy various QA systems to retrieve documents and evidence for each query.
  4. These retrieved items are then deduplicated and pooled.
  5. Human annotators (and later, GPT-4) evaluate the relevance of each piece of evidence to the corresponding query, establishing a ground truth for evaluation.
  6. System Evaluation:
    • The paper evaluates a diverse set of QA systems by systematically varying components like document retrievers (Wikipedia API vs. DuckDuckGo API), evidence indexers (text-only, textual-visual, modality-specific), evidence retrievers (sparse, dense, generative, hybrid), and LLMs.
  7. They also compare one-round (standard RAG) vs. multi-round (LLM Agent) retrieval strategies.
  8. Evaluation metrics include evidence relevancy, correct usage of evidence, and citation completeness.

Datasets

The primary contribution in terms of datasets is the introduction of OMG-QA, an open-domain, multi-modal generative question answering dataset.

  • OMG-QA: Contains 1,000 questions, with 500 generated from each of the two pipelines described above. Each question is linked to an average of 10 relevant documents and 33 pieces of relevant evidence. The dataset specifically targets scenarios requiring the integration of text, tables, and images.

Key Insights and Ideas

  • Multi-modal is Hard: The core insight is that current RAG systems struggle significantly when faced with the need to integrate information from multiple modalities (text, tables, images) and across multiple documents. OMG-QA highlights this challenge effectively.
  • Multi-index Wins: For this multi-modal dataset, systems using multiple indices (e.g., separate indices for text and images) consistently outperform single-index setups. This is because a single index struggles to efficiently represent and retrieve diverse modalities.
  • LLM Choice Matters Less Than Indexing/Retrieval: Surprisingly, the choice of LLM (even between smaller and much larger models) had a minimal impact on retrieval performance. This suggests that the architecture of the indexing and retrieval components plays a more crucial role.
  • Multi-round Retrieval's Double-Edged Sword: While multi-round retrieval (LLM agents) improves document recall, it can sometimes degrade evidence retrieval performance. The hypothesis is that the constant re-ranking in subsequent rounds might inadvertently discard relevant evidence.
  • GPT-4 as an Evaluator: The paper demonstrates a high agreement between human judgments and GPT-4 evaluations for tasks like evidence relevancy, correct evidence usage, and citation completeness, suggesting GPT-4's potential as a scalable evaluator.

Key Results

  • Document Retrieval: DuckDuckGo's search API consistently outperformed Wikipedia's own API for document retrieval across various LLM configurations.
  • Evidence Retrieval: Multi-index setups significantly improved evidence retrieval performance compared to text-only indices. Dense retrievers (especially those using models like snowflake-arctic-embed-1 and gte-large-en-v1.5) generally outperformed sparse and generative retrievers.
  • LLM Impact: The choice of LLM had a negligible impact on retrieval performance, with models like Llama-3-8b performing similarly to larger models.
  • Retrieval Strategy: Multi-round retrieval improved document recall but sometimes led to a decrease in evidence retrieval performance.
  • Best Performing System: The top-performing system combined DuckDuckGo for document retrieval, modality-specific indexing, gte-large-en-v1.5 for text/table, CLIP for images, and GPT-4 for LLM tasks, utilizing multi-round retrieval with memory and self-reflection.
  • Answer/Citation Quality: Proprietary models like GPT-3.5-Turbo and GPT-4 excelled in effective retrieval usage and citation completeness. GritLM-8x7b showed strong performance in the relevance and accuracy of attributed evidence.

Limitations and Future Work

  • Single Modality Solvability: Some questions in OMG-QA might be solvable using only a single modality, which could challenge the necessity of a strictly multi-modal approach for all questions.
  • GPT-4 Evaluation Staticness: GPT-4 evaluates evidence in isolation, lacking the dynamic, iterative adjustment that human annotators can perform when assessing relevance in the context of a multi-hop question. This static approach might lead to overlooking relevant evidence.
  • Future Work: The authors position OMG-QA as a benchmark to drive the development of more capable QA systems that can better handle open-domain, multi-modal information-seeking tasks. Future work would likely involve developing more robust multi-modal RAG architectures and exploring more effective multi-round retrieval strategies.

Summary updated

Abstract

We introduce OMG-QA, a new resource for question answering that is designed to evaluate the effectiveness of question answering systems that perform retrieval augmented generation (RAG) in scenarios that demand reasoning on multi-modal, multi-document contexts. These systems, given a user query, must retrieve relevant contexts from the web, which may include non-textual information, and then reason and synthesize these contents to generate a detailed, coherent answer. Unlike existing open-domain QA datasets, OMG-QA requires systems to navigate and integrate diverse modalities and a broad pool of information sources, making it uniquely challenging. We conduct a thorough evaluation and analysis of a diverse set of QA systems, featuring various retrieval frameworks, document retrievers, document indexing approaches, evidence retrieval methods, and LLMs tasked with both information retrieval and generation. Our findings reveal significant limitations in existing approaches using RAG or LLM agents to address open questions that require long-form answers supported by multi-modal evidence. We believe that OMG-QA will be a valuable resource for developing QA systems that are better equipped to handle open-domain, multi-modal information-seeking tasks.
BibTeX
@inproceedings{nan2024omg,
  author = {Linyong Nan and Weining Fang and Aylin Rasteh and Pouya Lahabi and Weijin Zou and Yilun Zhao and Arman Cohan},
  title = {{OMG-QA: Building Open-Domain Multi-Modal Generative Question Answering Systems}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=L0txLP8Aby},
}
Figure