TL;DR
This paper introduces LiteraryQA, a high-quality, human- and LLM-validated subset of the NarrativeQA benchmark, designed to address noise and inconsistencies in existing long-document narrative QA datasets. They also conduct a meta-evaluation of automatic metrics, finding that LLM-as-a-judge approaches, especially when provided with summaries, offer a more reliable evaluation of narrative QA systems than traditional n-gram-based metrics.
What Problem Does This Paper Address?
Evaluating Question Answering (QA) systems on long, narrative documents like books presents a significant challenge. Existing benchmarks, such as NarrativeQA, while widely adopted, suffer from noisy documents and flawed QA pairs, making it difficult to reliably assess model performance. This noise can lead to low correlations between automatic evaluation metrics and human judgment, and it's unclear whether poor performance is due to the inherent difficulty of the task or the dataset's quality. The authors aim to create a cleaner, more reliable benchmark and to understand which evaluation metrics are most effective for this challenging task.
Methods
The core contribution of this work is the creation of LiteraryQA, a refined subset of NarrativeQA. This was achieved through a two-phase data refinement pipeline:
- Document-level Phase: This involved filtering out non-book documents (e.g., theatrical plays, poetry collections) and cleaning the text by removing HTML artifacts, headers, footers, and legal notices. They also developed an automatic approach using LLMs to classify document types.
- QA-level Phase: This phase focused on individual question-answer pairs. It included:
- Question Deduplication: Removing redundant questions.
- Question Refinement: Using an LLM (Claude 3.5 Haiku) to identify and correct malformed or ill-posed questions based on provided summaries.
- Answer Refinement: Similarly, using an LLM to identify and correct malformed or invalid reference answers.
The authors also conducted a meta-evaluation of QA metrics. They compared traditional n-gram-based metrics (EM, F1, ROUGE-L, METEOR) and a neural metric (BERTScore) against human judgments on both the original NarrativeQA and the new LiteraryQA dataset. Crucially, they also evaluated LLMs (GPT-4.1, Claude 3.7 Sonnet, Prometheus 2.7B) as judges, testing them in both reference-based and summary-based settings.
Datasets
- NarrativeQA: The original dataset used as a starting point.
- LiteraryQA: The newly introduced, human- and LLM-validated subset of NarrativeQA, focused exclusively on literary works. This dataset aims to be more balanced, structurally, and stylistically homogeneous.
- BookCOREF: Mentioned as a resource for coreference resolution in literary texts.
- LitBank: Mentioned for annotations on literary event detection and entities.
Key Insights and Ideas
- Dataset Noise is a Major Hurdle: The significant improvements seen when moving from NarrativeQA to LiteraryQA highlight how much noise in the original dataset was hindering reliable evaluation.
- LLM-as-a-Judge is Promising: LLMs, especially when provided with the book summary, show a much higher correlation with human judgments than traditional n-gram-based metrics. This suggests they can better capture the nuances of narrative understanding.
- Summary-Based Evaluation is Key: Providing LLM judges with the book summary significantly boosts their correlation with human judgments, as it allows them to disregard potentially noisy reference answers and focus on the overall narrative context.
- N-gram Metrics Struggle with Narrative QA: Traditional metrics like EM and F1 have very low correlations with human judgments, indicating they are not well-suited for abstractive, long-document narrative QA. METEOR performs slightly better but still lags behind LLM-based evaluations.
- LiteraryQA is a More Reliable Benchmark: The improved quality of LiteraryQA leads to higher scores for tested models across all metrics and better reflection of human judgments, making it a more suitable benchmark for evaluating narrative QA systems.
Key Results
- Data Refinement Impact: The LiteraryQA pipeline removed 22% of documents from the original NarrativeQA test set and modified 38% of QA samples. On average, documents were reduced by 3K tokens (12%).
- Metric Correlation:
- N-gram-based metrics showed poor correlations with human judgments on NarrativeQA (except METEOR).
- On LiteraryQA, n-gram metrics showed slightly better correlations, indicating the dataset's improved quality. METEOR achieved a system-level correlation of 0.44.
- LLM-as-a-judge models showed significantly higher correlations. In the summary-based setting, Prometheus 2.7B achieved a correlation of 0.68 on LiteraryQA.
- LLM Performance: When benchmarked on LiteraryQA, open-weight models like NExtLong-8B and Qwen2.5-14B performed well on n-gram metrics, but LLM-as-a-judge evaluations revealed that closed-source models like Claude 3.5 Haiku often performed better, especially in summary-based settings.
- Model Performance on LiteraryQA: Models generally performed better on LiteraryQA than on the original NarrativeQA, with LLM-as-a-judge metrics showing a particularly large gap, suggesting LiteraryQA is a more sensitive and reliable benchmark.
Limitations and Future Work
- LLM Bias: The data refinement pipeline relies partly on LLMs, which could introduce potential biases, although human oversight mitigates this to some extent.
- Literary Focus: LiteraryQA exclusively focuses on literary works, excluding other narrative forms like movie scripts or plays, limiting its representativeness of the full narrative landscape.
- No RAG Evaluation: The study did not include retrieval-augmented generation (RAG) approaches, as the focus was on pure comprehension and reasoning over entire texts. Exploring RAG in this setting is suggested for future work.
- LLM-as-a-Judge Cost and Transparency: LLM-as-a-judge evaluations are costly and lack transparency, posing challenges for reproducibility. While small fine-tuned models like Prometheus show promise, the authors suggest that models specifically fine-tuned for narrative evaluation could offer more accurate and cost-effective alternatives.
- Future Work: The authors suggest further research into LLM-as-a-judge approaches, particularly exploring summary-based evaluations and developing models specifically for narrative QA. They also note that the open-ended narrative QA setting remains a challenging benchmark.
Summary updated