MDCR: A Dataset for Multi-Document Conditional Reasoning

Peter Baile Chen, Yi Zhang, Chunwei Liu, Sejal Gupta, Yoon Kim, Michael J. CafarellaarXiv 2024 · DOI · OpenReview

TL;DR

Large language models (LLMs) struggle with complex reasoning tasks that require understanding and combining information from multiple documents, especially when optimization is involved. To address this, researchers introduce MDCR, a new dataset for multi-document conditional reasoning, and demonstrate that current LLMs perform poorly on this challenging task, highlighting the need for further research.

What Problem Does This Paper Address?

In real life, many questions, like scholarship eligibility, depend on specific conditions that aren't always explicitly stated. While previous work (like ConditionalQA) tackled answering questions based on single documents with unmentioned conditions, it didn't cover scenarios where users need to find the best possible outcome by considering information across multiple documents. For example, a student might want to know the maximum number of scholarships they can receive. This requires not only understanding conditions within each document but also how conditions relate across documents (e.g., conflicting, equivalent, or inclusive) and then optimizing to find the best combination. This multi-document, optimization-focused conditional reasoning is a significant gap in current AI capabilities.

Methods

The core of this work is the creation of a new dataset, MDCR (Multi-Document Conditional Reasoning). The process involved:

  1. Document Collection: Gathering HTML documents from scholarship and job application domains, which typically contain eligibility conditions.
  2. Human Annotation: Annotators extracted eligibility conditions, identified AND/OR relationships within documents, and labeled relationships (conflicting, equivalent, inclusive) between conditions from different documents.
  3. Scenario Generation: User scenarios were created to mimic real-world situations, including both relevant and irrelevant information. These scenarios were designed to encourage reasoning about condition relationships.
  4. Gold Answer Generation: A symbolic solver was used to automatically generate correct answers for the dataset, leveraging the annotated conditions and relationships.

The paper then uses this dataset to benchmark existing LLMs.

Datasets

The paper introduces the MDCR (Multi-Document Conditional Reasoning) dataset. It consists of:

  • Documents: From scholarship and job application domains.
  • User Scenarios: Descriptions of user backgrounds.
  • User Questions: Three types:
  • Q1: Can I receive at least one outcome?
  • Q2: Can I receive all outcomes?
  • Q3: What is the maximum number of outcomes I can receive?
  • Outputs: Short answers (yes/no or a number) and conditional answers (lists of unmentioned conditions required for the answer).

MDCR differs from previous datasets by focusing on multi-document reasoning and optimization, which are more complex than single-document eligibility checks.

Key Insights and Ideas

  • Multi-document reasoning is hard: LLMs struggle significantly when they need to synthesize information from multiple sources to answer conditional questions, especially when optimization is involved.
  • Optimization is a major hurdle: Questions asking for the maximum number of outcomes (Q3) are considerably more challenging than simple eligibility checks (Q1) or checking if all outcomes are possible (Q2).
  • Condition relationships are crucial: The ability to understand and leverage conflicting, equivalent, and inclusive relationships between conditions across documents is essential for accurate reasoning, and LLMs often fail here.
  • Context length matters: Performance degrades as the number of documents increases, suggesting that LLMs have difficulty processing and reasoning over longer contexts.
  • LLMs are better at short answers than conditional details: Models can often get the "yes/no" or numerical answer right, but providing the precise set of unmentioned conditions that justify that answer is much more difficult.

Key Results

  • Baseline Performance: Even the best-performing LLM (GPT-4o) achieved only around 69% accuracy on short answers and a mere 40% F1 score for conditional answers in the multi-document setting.
  • Question Type Difficulty: For Q3 (maximization), short answer accuracy dropped to an average of 46.1%, and conditional answer F1 to 7.84%.
  • Impact of Document Count: Performance dropped significantly when moving from single-document to multi-document settings, with short answer accuracy decreasing by an average of 5.9% and conditional answer F1 by 23.1%.
  • Effect of Gold Information: Providing LLMs with hints about condition extraction, satisfiability, or relationships significantly improved performance, demonstrating their reliance on explicit guidance. However, even with all hints, performance remained imperfect.

Limitations and Future Work

  • Domain Specificity: The current dataset focuses on scholarships and job applications. Future work could expand to other domains and question types.
  • External Knowledge: The paper acknowledges that real-world reasoning might require external knowledge (e.g., about citizenship laws) that is not explicitly provided in the documents. Investigating how LLMs can leverage such external knowledge for conditional reasoning is a promising direction.
  • Reasoning Depth: While LLMs can be guided with hints, their core reasoning capabilities for complex conditional logic and optimization remain limited, indicating a need for architectural or training improvements.

Summary updated

Abstract

The same real-life questions posed to different individuals may lead to different answers based on their unique situations. For instance, whether a student is eligible for a scholarship depends on eligibility conditions, such as major or degree required. ConditionalQA was proposed to evaluate models' capability of reading a document and answering eligibility questions, considering unmentioned conditions. However, it is limited to questions on single documents, neglecting harder cases that may require cross-document reasoning and optimization, for example, "What is the maximum number of scholarships attainable?" Such questions over multiple documents are not only more challenging due to more context having to understand, but also because the model has to (1) explore all possible combinations of unmentioned conditions and (2) understand the relationship between conditions across documents, to reason about the optimal outcome. To evaluate models' capability of answering such questions, we propose a new dataset MDCR, which can reflect real-world challenges and serve as a new test bed for complex conditional reasoning that requires optimization. We evaluate this dataset using the most recent LLMs and demonstrate their limitations in solving this task. We believe this dataset will facilitate future research in answering optimization questions with unknown conditions.
BibTeX
@misc{chen2024mdcr,
  author = {Peter Baile Chen and Yi Zhang and Chunwei Liu and Sejal Gupta and Yoon Kim and Michael J. Cafarella},
  title = {{MDCR: A Dataset for Multi-Document Conditional Reasoning}},
  year = {2024},
  eprint = {2406.11784},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.11784},
}
Figure