S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models

Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, Kang Liu

Introduction

As Large Language Models (LLMs) expand their context windows to 200K tokens and beyond, a significant "evaluation gap" has emerged. Current benchmarks often fail to distinguish between simple retrieval and genuine reasoning over long contexts. Popular tests like "Needle-in-a-Haystack" primarily measure basic fact retrieval, while real-world datasets like WikiTableQuestions are difficult to scale and prone to data leakage. S3Eval introduces a synthetic benchmark using SQL execution as a proxy to rigorously evaluate reasoning and long-context capabilities without these drawbacks.

Methodology: SQL Execution as a Reasoning Proxy

S3Eval leverages the structured nature of SQL to create complex, multi-step logic tasks that are easy to verify but difficult for models to solve without true synthesis. The framework consists of three main control components:

  1. Table Control: Generates tables with adjustable dimensions, filled with random data to prevent leakage.
  2. Instruction Control: Uses a context-free grammar to generate diverse SQL queries involving filtering (WHERE), aggregation (COUNT, MAX), and sorting (ORDER BY).
  3. Output Control: Manages variables like answer location and distance to test specific model behaviors.
The S3Eval framework systematically controls table generation and SQL instructions to evaluate LLM reasoning and long-context understanding.

Validation and Key Insights

A critical finding of this work is that SQL execution performance serves as an excellent proxy for real-world reasoning. The authors observed a near-perfect correlation ($r=99.1$) between a model's ability to execute S3Eval queries and its performance on the WikiTableQuestions benchmark. This validates that if a model cannot handle the logic of SQL, it likely cannot handle complex real-world reasoning.

S3Eval SQL execution scores show a high correlation with performance on real-world table-based question answering tasks.

Beyond validation, S3Eval revealed several nuances in LLM behavior:

  • Dense vs. Sparse Bottlenecks: Models struggle significantly more when relevant data points are scattered across a table ("Sparse") compared to when they are adjacent ("Dense").
  • Fine-Grained "Lost in the Middle": The benchmark confirms that performance degradation occurs at a token level, often following a "wave-like" pattern related to position embeddings.
  • Structural Sensitivity: LLMs perform better with "flattened" table formats that repeat column names, suggesting they rely on redundant structural cues in long contexts.

Performance Results

Evaluation of leading models revealed a stark performance gap as context length increases. GPT-3.5-Turbo, for instance, saw accuracy drop from ~40% to ~16% when moving from short to long contexts. GPT-4-32K emerged as the top performer with a 54.8% overall score, while Claude-1.3-100K demonstrated the highest stability across varying context lengths. These results suggest that even state-of-the-art models still struggle to maintain reasoning consistency as context windows expand.

Conclusion

S3Eval provides a scalable and systematic way to probe the limits of long-context LLMs. By moving from simple retrieval to complex synthetic reasoning, it offers a more robust measure of model utility in data-heavy applications. Future work will focus on expanding query complexity and multilingual support.

Summary updated

Abstract

The rapid development of Large Language Models (LLMs) has led to great strides in model capabilities like reasoning and long-context understanding. However, as LLMs are able to process longer contexts, it becomes more challenging to evaluate whether they have acquired certain capabilities, since the length of text (e.g., 100K tokens) they can process far exceeds what humans can reliably assess in a reasonable duration. In this paper, we propose using complex synthetic tasks as a proxy evaluation method, and present S3Eval, a Synthetic, Scalable, Systematic evaluation suite for LLMs evaluation. As a synthetic benchmark, S3Eval enables the creation of any number of evaluation examples that are theoretically invisible to LLMs, mitigating the test set contamination issue. The synthetic nature of S3Eval provides users full control over the dataset, allowing them to systematically probe LLM capabilities by scaling text length and varying task difficulty across diverse scenarios. The strong correlation between S3Eval performance and scores of real-world benchmarks like Big-Bench Hard (BBH) demonstrates the soundness of using S3Eval for evaluation of LLMs. The in-depth analysis also uncover additional insights, including performance drop when the answer is sparsely distributed or located in the middle context, as well as some counter-intuitive trends of model performance.
BibTeX
@misc{lei2023s,
  author = {Fangyu Lei and Qian Liu and Yiming Huang and Shizhu He and Jun Zhao and Kang Liu},
  title = {{S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models}},
  year = {2023},
  eprint = {2310.15147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.15147},
}
Figure