S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models

Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, Kang Liu

TL;DR

S3Eval is a new synthetic evaluation suite that uses SQL execution tasks to rigorously test the reasoning and long-context capabilities of Large Language Models (LLMs). By generating infinite, verifiable, and complex test cases, it provides a scalable alternative to human-annotated benchmarks and simple retrieval tasks like "Needle-in-a-Haystack."

What Problem Does This Paper Address?

As LLMs evolve to support massive context windows—sometimes exceeding 200,000 tokens—evaluating them becomes a logistical nightmare. Human annotation for such lengths is prohibitively expensive and slow. Furthermore, existing benchmarks often fall into two traps:

  1. Simplicity: Popular tests like "Needle-in-a-Haystack" only measure simple retrieval, failing to capture the sophisticated reasoning required for real-world applications.
  2. Data Leakage: Many benchmarks use public datasets that likely appeared in the models' training corpora, leading to inflated performance scores.

The authors address these gaps by proposing a Synthetic, Scalable, and Systematic (S3) approach that uses SQL execution as a proxy for complex reasoning.

Methods

The core of S3Eval is the automated generation of random tables and corresponding SQL queries.

Figure 2: The illustration demonstrates the S3EVAL pipeline, where the capabilities of LLMs are assessed by evaluating their ability to execute SQL queries over randomly generated tables.

The framework gives researchers granular control over three areas:

  • Table Control: Adjusting the number of rows/columns, data types (INT, TEXT, DATE), and value uniqueness.
  • Instruction Control: Tuning SQL complexity by selecting specific operators (JOIN, GROUP BY, HAVING), nesting depth, and the number of required calculations.
  • Output Control: Precisely placing the "answer" tokens within the context to test for positional biases.

Because the tasks are synthetic, the "ground truth" is determined by a standard SQL engine, allowing for infinite data generation without human intervention.

Datasets

The authors introduced S3Eval-Standard, a diverse benchmark covering context lengths from 2K to 40K tokens. To validate S3Eval, they compared model performance against several established benchmarks:

  • WikiTableQuestions: For table-based question answering.
  • BBH (Big-Bench Hard): For general symbolic and logical reasoning.
  • HumanEval: For coding and programming logic.

Key Insights and Ideas

  • SQL as a Robust Proxy: The authors found a remarkably high correlation between a model's ability to execute synthetic SQL and its performance on "real-world" reasoning tasks. If a model is good at S3Eval, it’s generally good at BBH and HumanEval.
  • The "Lost in the Middle" Phenomenon: S3Eval allows for token-level precision in answer placement. The results confirm that LLMs struggle significantly when the information needed to solve a task is buried in the middle of a long prompt, rather than at the beginning or end.
  • Dense vs. Sparse Information: Models perform much better when the relevant data points are "dense" (adjacent rows) rather than "sparse" (scattered across the document). This suggests that current long-context models still struggle with global dependency modeling.
  • Scaling Laws Apply: The authors demonstrated that S3Eval scores improve predictably with increased training compute, aligning with established scaling laws (see Figure 3).

Key Results

The experimental results show that S3Eval is significantly more challenging than simple retrieval tasks. While many models achieve near 100% on "Needle-in-a-Haystack," their performance craters on S3Eval as context length grows.

Figure 4: The performance of different LLMs on S3EVAL and WikiTableQuestions.
  • Performance Cliff: Even top-tier models like GPT-4 saw a drop from 68.4% accuracy on short contexts to 43.0% on contexts up to 40K tokens.
  • High Correlation: S3Eval showed a Pearson correlation coefficient (r) of 95.3 with the BBH benchmark and 99.1 with WikiTableQuestions, proving its validity as a proxy for general reasoning.
Figure 5: Each point in the scatterplot represents the LLM performance on the benchmarks corresponding to the horizontal and vertical coordinates. The black straight line is the trend line. The larger the values of r and τ, the higher the correlation between the two benchmarks. We consider τ > 0.8 to be high concurrence.

Limitations and Future Work

While S3Eval is a powerful tool, the authors note a few areas for improvement:

  • SQL Diversity: The current grammar for generating SQL queries is still somewhat limited and could be expanded to include even more complex operations.
  • Multilingualism: While the framework supports multilingual testing in theory, the paper does not provide a systematic analysis of LLM performance across different languages.
  • Natural Language Gap: While SQL is a great proxy, it remains a symbolic language. Future work could explore bridging these synthetic tasks more closely with natural language reasoning patterns.

In conclusion, S3Eval offers a rigorous, leak-proof, and infinitely scalable way to "stress test" the next generation of long-context LLMs.

Summary updated

Abstract

The rapid development of Large Language Models (LLMs) has led to great strides in model capabilities like long-context understanding and reasoning. However, as LLMs are able to process longer contexts, it becomes more challenging to evaluate whether they have acquired certain capabilities, since the length of text (e.g., 200K tokens) they can process far exceeds what humans can reliably assess in a reasonable duration. In this paper, we propose using complex synthetic tasks as a proxy evaluation method, and present S3Eval, a Synthetic, Scalable, Systematic evaluation suite for LLMs evaluation. The synthetic nature of S3Eval provides users full control over the dataset, allowing them to systematically probe LLM capabilities by scaling text length and varying task difficulty across diverse scenarios. The strong correlation between S3Eval and real-world benchmarks demonstrates the soundness of using S3Eval for evaluation of LLMs. S3Eval provides a flexible and infinite long-context data generation method. We have generated a comprehensive dataset called S3Eval-Standard, and experimental results have shown that it poses significant challenges for all existing LLMs.
BibTeX
@misc{lei2023s,
  author = {Fangyu Lei and Qian Liu and Yiming Huang and Shizhu He and Jun Zhao and Kang Liu},
  title = {{S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models}},
  year = {2023},
  eprint = {2310.15147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.15147},
}
Figure