Introduction
Large Language Models (LLMs) are increasingly deployed for tasks requiring the synthesis of information from multiple sources, such as multi-document summarization and cross-document coreference resolution. However, these models are notoriously sensitive to minor prompt variations. Swapping the order of input documents or slightly altering instruction phrasing can lead to significantly different outputs. This "finickiness" undermines the reliability of static benchmarks that report a single performance score, as a model's success might be the result of a "lucky" prompt configuration rather than underlying capability.
The SEAM Framework
The authors introduce SEAM (Stochastic Evaluation Approach for Multi-document tasks), a benchmark designed to measure model performance while accounting for inherent prompt volatility. Instead of using a fixed set of prompts, SEAM acts as a sample generator that introduces randomness into five "unrelated" variables: instruction paraphrasing, document ordering, few-shot example selection, few-shot ordering, and instance sampling.
By executing multiple iterations of this randomized process, SEAM provides a distribution of scores (mean and standard deviation). This approach identifies models that are truly robust versus those that simply perform well on specific, static prompt instances.
Experimental Setup
SEAM aggregates six datasets across three task families to test the limits of multi-document reasoning:
- Summarization: MultiNews (news articles), OpenAsp (aspect-based news), and FuseReviews (conflicting business reviews).
- Multi-hop QA: MuSiQue (reasoning across Wikipedia articles).
- Cross-Document Coreference: ECB+ (news) and SciCo (scientific papers).
The benchmark was used to evaluate several open-weight model families, including Llama 3 (8B, 70B), Mistral/Mixtral, and Gemma 1.1 (2B, 7B).
Key Findings and Model Performance
The stochastic evaluation revealed that even state-of-the-art models struggle with the unique challenges of multi-document consolidation.
High Sensitivity to Prompt Noise
A key finding is the significant variance in model performance across different stochastic iterations. As shown in the results for the MultiNews dataset, models exhibit noticeable spreads in their ROUGE scores. This variance highlights the danger of relying on single-point evaluations and demonstrates that even powerful models like Llama 3-70B are not immune to prompt noise.
Consolidation vs. Context Length
The study found that a model's context window size does not necessarily correlate with its ability to synthesize information. The primary bottleneck is "consolidation"—the cognitive task of resolving redundancies and contradictions across documents—rather than simple memory or attention span. Simply having a 128k context window does not guarantee a model can effectively navigate the messy nature of multi-source data.
Robustness and Model Size
Interestingly, larger models are not always more robust. In several instances, smaller models like Mistral-7B demonstrated lower relative standard deviation or even higher absolute performance than larger counterparts like Llama 3-70B in specific summarization tasks. Furthermore, a model's ability to follow complex output formats (e.g., JSON) often serves as a primary bottleneck for overall task success; if a model fails the format check, it fails the task regardless of its reasoning.
Conclusion
SEAM offers a more rigorous and honest framework for evaluating LLMs on complex multi-document tasks. By embracing stochasticity, it reveals that current models are far from solving these challenges and remain highly sensitive to prompt engineering. Future research should focus on improving the "consolidation" capabilities of LLMs and establishing human-level baselines to quantify the remaining performance gap.
Summary updated