seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs

Mohammad Ramezanali, Mo Vazifeh, Paolo SantiEMNLP 2025

TL;DR

Large Language Models (LLMs) excel at many tasks but struggle with complex sequential reasoning. This paper introduces seqBench, a novel benchmark that precisely controls key reasoning challenges like logical depth and noise, revealing that even state-of-the-art LLMs exhibit sharp performance cliffs as reasoning complexity increases. This work provides a crucial tool for understanding and improving LLM reasoning capabilities.

What Problem Does This Paper Address?

LLMs have demonstrated impressive capabilities across a wide range of tasks, often leading to the perception of near-human-level intelligence. However, these successes can mask fundamental limitations, particularly in sequential reasoning. This type of reasoning, which involves a series of logical steps and dependencies, is crucial for tasks requiring planning, problem-solving, and understanding complex narratives. Existing benchmarks often fail to isolate specific reasoning challenges, making it difficult to pinpoint why LLMs falter. This lack of fine-grained control prevents systematic studies into the precise factors that limit LLM reasoning, such as the length of the reasoning chain, the need to revisit previous states (backtracking), or the presence of distracting information. Without this understanding, it's hard to develop more robust and reliable AI systems.

Methods

The core of this paper is the introduction of seqBench, a tunable benchmark designed to systematically probe LLM sequential reasoning. The benchmark generates synthetic spatial pathfinding tasks on 2D grids. The novelty lies in its ability to precisely control three key complexity dimensions:

  1. Logical Depth (L): The number of sequential actions required to solve the task.
  2. Backtracking Count (B): The number of times an agent must revisit a state to satisfy a deferred precondition (e.g., retrieving a key for a locked door).
  3. Noise Ratio (N): The proportion of distracting facts relative to supporting facts in the task description.

seqBench uses a "Rewind Construction" method to generate these tasks. This involves starting from a goal and working backward to strategically place keys and locked doors, ensuring solvability and controlled backtracking. The generated tasks are then translated into natural language facts. The benchmark is designed to be scalable and evolvable, allowing it to adapt to increasingly capable LLMs.

Datasets

The paper introduces seqBench, a newly generated dataset of synthetic spatial pathfinding tasks. This dataset is publicly available and comprises 7,079 curated instances. Unlike existing benchmarks, seqBench offers fine-grained, orthogonal control over logical depth, backtracking, and noise ratio, enabling targeted analysis of LLM reasoning failures. The generation process allows for systematic variation of grid sizes, target backtracking steps, and noise ratios.

Key Insights and Ideas

  • Universal Performance Cliffs: All evaluated LLMs, regardless of their size or family, exhibit a sharp, exponential decline in performance as logical depth (the length of the reasoning chain) increases. This suggests a fundamental architectural bottleneck in current LLMs for sustained multi-step reasoning.
  • Backtracking is a Major Hurdle: Increasing the number of required backtracking steps significantly degrades LLM performance, indicating a struggle with managing dependencies and revisiting intermediate conclusions.
  • Noise is Detrimental: The presence of distracting information (noise ratio) is highly sensitive and significantly degrades LLM accuracy, even when the underlying logical structure remains simple.
  • Fact Ordering Matters Less: In contrast to noise and backtracking, the order in which facts are presented (shuffle ratio) has a minimal impact on performance, suggesting LLMs are relatively robust to presentation order as long as the information is present.
  • Early Failures in Complex Tasks: As problem complexity (logical depth) increases, LLMs tend to fail earlier in the reasoning sequence. This suggests that anticipated global complexity, rather than just cumulative errors, impacts reasoning from the outset.
  • Disparity in Capacity vs. Reasoning: Modern LLMs possess vast context windows (millions of tokens), yet their effective sequential reasoning depth is limited to hundreds of actions. This highlights a significant gap between information retention and the ability to chain information for coherent inference.

Key Results

  • Logical Depth (L): Performance collapses exponentially beyond a model-specific logical depth ($L_0$). For example, Gemini-2.5-Flash has an $L_0$ of 85.7, while Llama-3.2-3B has an $L_0$ of 1.6, indicating a wide range in reasoning horizons.
  • Backtracking: Increasing backtracking steps from 0 to 5 leads to a clear decline in Pass@1 success rates and mean progress ratios across all models.
  • Noise Ratio (N): For Gemini 2.5 Flash, Pass@1 success drops from over 0.7 at zero noise to approximately 0.2 at a noise ratio of 1.0.
  • Failure Modes: Models predominantly fail by omitting critical sub-goals or steps (low recall), rather than hallucinating actions (high precision).

Limitations and Future Work

  • Generalizability and Task Design Fidelity: The current findings are based on synthetic spatial pathfinding tasks. Future work should extend seqBench to other reasoning domains (e.g., mathematical proofs) and incorporate linguistic diversity and visual modalities to assess broader applicability.
  • Model Scope and Deeper Failure Dynamics: The evaluation should be expanded to include more LLMs, especially recent proprietary and open-source variants. Further research is needed to understand why $L_0$ is influenced by factors like model architecture, scale, training data, and inference strategies.
  • Agentic Systems and Tool Use: The current evaluation does not consider how agentic systems capable of tool use perform. Exploring setups where LLMs can externalize sub-problems or invoke tools could offer valuable insights.
  • Impact of Prompting: A robust sensitivity analysis is needed to determine how different prompting strategies (e.g., few-shot, chain-of-thought) influence decay behavior and whether they can mitigate observed failures.

Summary updated

BibTeX
@inproceedings{ramezanali2025seqbench,
  author = {Mohammad Ramezanali and Mo Vazifeh and Paolo Santi},
  title = {{seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure