SportReason: Evaluating Retrieval-Augmented Reasoning across Tables and Text for Sports Question Answering

Kaiyue Feng, Siyue Zhang, Bingqiu Chen, Yilun Zhao, Zhao Chen2025 · DOI

Abstract

We present SPORTREASON, a benchmark for retrieval-augmented reasoning on numerical sports questions. Unlike existing benchmarks limited to one or two evidence units, SPORTREASON requires combining and reasoning across free-text, structured tables, and semi-structured infoboxes. We provide 3,000 human-verified QA pairs by repurposing existing QA and table generation datasets, and by prompting large language models (LLMs). Each pair is grounded in multiple evidence from a multi-modal Wikipedia corpus containing 200K knowledge contexts. We evaluate existing retrievers and rerankers, along with agentic Retrieval-Augmented Generation (RAG) systems. The experimental results show that multi-evidence retrieval remains a challenge. Agentic RAG systems (e.g., Search-o1), despite iterative retrieval and reasoning capabilities, fail to improve performance due to imprecise query generation and distracting retrieval information.

Introduction: The Challenge of Multi-Modal Sports Reasoning

Retrieval-Augmented Generation (RAG) has become the standard for providing Large Language Models (LLMs) with "open-book" access to external knowledge. However, most existing benchmarks evaluate models on relatively simple tasks, such as finding an answer within a single paragraph or a single table. In reality, complex questions—particularly in the sports domain—often require synthesizing information from fragmented sources.

Consider a question like: "What was the highest number of races held in a Formula One season when Lewis Hamilton was the Drivers' Champion?" Answering this requires identifying the years Hamilton won the championship (often found in an infobox or text summary) and then cross-referencing those years with a table of season statistics to find the race counts. This necessitates multi-modal, multi-evidence reasoning across text, tables, and infoboxes.

Overview of the SportReason task requiring multi-modal reasoning.

The SportReason Benchmark

To address the lack of challenging multi-modal benchmarks, we introduce SportReason. This dataset is specifically designed to test an AI's ability to aggregate numerical data across diverse data structures. The benchmark consists of 3,000 human-verified QA pairs curated from existing datasets like HybridQA, TANQ, and TempTabQA, and augmented through LLM generation. These are grounded in a massive multi-modal corpus of 200,000 Wikipedia documents comprising text passages, structured tables, and semi-structured infoboxes.

Methodology

We evaluated a variety of RAG architectures on the SportReason benchmark:

  1. Retrieval & Reranking: We tested a spectrum of models, ranging from traditional keyword-based BM25 to state-of-the-art LLM-based embedding models such as INF-Retriever and BGE-Gemma2.
  2. Reasoning Paradigms: We compared "Vanilla RAG" (single-step retrieval) against Agentic RAG systems like Search-o1 and IRCoT. These agentic systems employ iterative planning and searching to gather information.
  3. Reader Model: To isolate the impact of retrieval and reasoning strategies, Gemini 2.5 Flash was utilized as the primary "reader" model for synthesizing final answers.

Results and Discussion: The Modality Gap

Our experiments revealed several critical insights into the current state of RAG systems.

The "Text Distraction" Effect

A primary finding is the significant performance degradation when models must combine tabular data with free-form text. While models perform relatively well on purely tabular tasks, their retrieval success plummets when relevant information is also embedded in prose. This "modality gap" suggests that current retrievers are easily distracted by irrelevant textual information when searching for specific numerical data.

Performance comparison across different evidence modalities, showing the drop in accuracy when text is added to tabular queries.

Retrieval as the Primary Bottleneck

Error analysis indicates that 70% of failures are attributable to retrieval issues—specifically, the failure to surface the necessary evidence—rather than reasoning errors. If the retriever fails to provide the correct data points, even the most capable LLM cannot generate the correct answer. Furthermore, "agentic" systems that can perform multiple searches did not significantly outperform vanilla RAG, as they often struggled with precise query generation for numerical tasks.

Conclusion and Future Directions

SportReason highlights that multi-evidence retrieval remains a formidable challenge for AI. While LLM-based retrievers show promise, they still struggle with the transition between structured and unstructured data. Future research should focus on developing more robust query planning strategies that can navigate multi-modal environments without being sidelined by distracting information.

Summary updated

Abstract

We present SPORTREASON, a benchmark for retrieval-augmented reasoning on numerical sports questions.Unlike existing benchmarks limited to one or two evidence units, SPORTREASON requires combining and reasoning across free-text, structured tables, and semi-structured infoboxes.We provide 3,000 human-verified QA pairs by repurposing existing QA and table generation datasets, and by prompting large language models (LLMs).Each pair is grounded in multiple evidence from a multi-modal Wikipedia corpus containing 200K knowledge contexts.We evaluate existing retrievers and rerankers, along with agentic Retrieval-Augmented Generation (RAG) systems.The experimental results show that multi-evidence retrieval remains a challenge.Agentic RAG systems (e.g., Search-o1), despite iterative retrieval and reasoning capabilities, fail to improve performance due to imprecise query generation and distracting retrieval information.
BibTeX
@misc{feng2025sportreason,
  author = {Kaiyue Feng and Siyue Zhang and Bingqiu Chen and Yilun Zhao and Zhao Chen},
  title = {{SportReason: Evaluating Retrieval-Augmented Reasoning across Tables and Text for Sports Question Answering}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.34},
}
Figure