SEAM: A Stochastic Benchmark for Multi-Document Tasks

Gili Lior, Avi Caciularu, Arie Cattan, Shahar Levy, Ori Shapira, Gabriel StanovskyarXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) are increasingly deployed for tasks requiring the synthesis of information from multiple sources, such as multi-document summarization and cross-document coreference resolution. However, these models are notoriously sensitive to minor prompt variations. Swapping the order of input documents or slightly altering instruction phrasing can lead to significantly different outputs. This "finickiness" undermines the reliability of static benchmarks that report a single performance score, as a model's success might be the result of a "lucky" prompt configuration rather than underlying capability.

The SEAM Framework

The authors introduce SEAM (Stochastic Evaluation Approach for Multi-document tasks), a benchmark designed to measure model performance while accounting for inherent prompt volatility. Instead of using a fixed set of prompts, SEAM acts as a sample generator that introduces randomness into five "unrelated" variables: instruction paraphrasing, document ordering, few-shot example selection, few-shot ordering, and instance sampling.

The SEAM evaluation pipeline: Multi-document tasks are processed through stochastic prompt configurations before being fed to an LLM to generate a distribution of results.

By executing multiple iterations of this randomized process, SEAM provides a distribution of scores (mean and standard deviation). This approach identifies models that are truly robust versus those that simply perform well on specific, static prompt instances.

Experimental Setup

SEAM aggregates six datasets across three task families to test the limits of multi-document reasoning:

  • Summarization: MultiNews (news articles), OpenAsp (aspect-based news), and FuseReviews (conflicting business reviews).
  • Multi-hop QA: MuSiQue (reasoning across Wikipedia articles).
  • Cross-Document Coreference: ECB+ (news) and SciCo (scientific papers).

The benchmark was used to evaluate several open-weight model families, including Llama 3 (8B, 70B), Mistral/Mixtral, and Gemma 1.1 (2B, 7B).

Key Findings and Model Performance

The stochastic evaluation revealed that even state-of-the-art models struggle with the unique challenges of multi-document consolidation.

High Sensitivity to Prompt Noise

A key finding is the significant variance in model performance across different stochastic iterations. As shown in the results for the MultiNews dataset, models exhibit noticeable spreads in their ROUGE scores. This variance highlights the danger of relying on single-point evaluations and demonstrates that even powerful models like Llama 3-70B are not immune to prompt noise.

Distribution of ROUGE scores for various LLMs on the MultiNews task, illustrating the performance variance and lack of saturation captured by the SEAM benchmark.

Consolidation vs. Context Length

The study found that a model's context window size does not necessarily correlate with its ability to synthesize information. The primary bottleneck is "consolidation"—the cognitive task of resolving redundancies and contradictions across documents—rather than simple memory or attention span. Simply having a 128k context window does not guarantee a model can effectively navigate the messy nature of multi-source data.

Robustness and Model Size

Interestingly, larger models are not always more robust. In several instances, smaller models like Mistral-7B demonstrated lower relative standard deviation or even higher absolute performance than larger counterparts like Llama 3-70B in specific summarization tasks. Furthermore, a model's ability to follow complex output formats (e.g., JSON) often serves as a primary bottleneck for overall task success; if a model fails the format check, it fails the task regardless of its reasoning.

Conclusion

SEAM offers a more rigorous and honest framework for evaluating LLMs on complex multi-document tasks. By embracing stochasticity, it reveals that current models are far from solving these challenges and remain highly sensitive to prompt engineering. Future research should focus on improving the "consolidation" capabilities of LLMs and establishing human-level baselines to quantify the remaining performance gap.

Summary updated

Abstract

Various tasks, such as summarization, multi-hop question answering, or coreference resolution, are naturally phrased over collections of real-world documents. Such tasks present a unique set of challenges, revolving around the lack of coherent narrative structure across documents, which often leads to contradiction, omission, or repetition of information. Despite their real-world application and challenging properties, there is currently no benchmark which specifically measures the abilities of large language models (LLMs) on multi-document tasks. To bridge this gap, we present SEAM (a Stochastic Evaluation Approach for Multi-document tasks), a conglomerate benchmark over a diverse set of multi-document datasets, setting conventional evaluation criteria, input-output formats, and evaluation protocols. In particular, SEAM addresses the sensitivity of LLMs to minor prompt variations through repeated evaluations, where in each evaluation we sample uniformly at random the values of arbitrary factors (e.g., the order of documents). We evaluate different LLMs on SEAM finding that multi-document tasks pose a significant challenge for LLMs, even for state-of-the-art models with 70B parameters. In addition, we show that the stochastic approach uncovers underlying statistical trends which cannot be observed in a static benchmark. We hope that SEAM will spur progress via consistent and meaningful evaluation of multi-document tasks.
BibTeX
@misc{lior2024seam,
  author = {Gili Lior and Avi Caciularu and Arie Cattan and Shahar Levy and Ori Shapira and Gabriel Stanovsky},
  title = {{SEAM: A Stochastic Benchmark for Multi-Document Tasks}},
  year = {2024},
  eprint = {2406.16086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.16086},
}
Figure