CMT-Bench: Cricket Multi-Table Generation Benchmark for Probing Robustness in Large Language Models

Ritam Upadhyay, Naman Ahuja, Rishabh Baral, Aparna Garimella, Vivek Gupta

Abstract

LLM Driven text-to-table (T2T) systems often rely on extensive prompt-engineering or iterative event extraction in code-parsable formats, which boosts scores but are computationally expensive and obscure how models actually reason over temporal evolving narratives to summarise key information. We present CMT-Bench, a diagnostic benchmark built from live cricket commentary that requires dynamic table generation across two evolving schemas under a dense, rule-governed policy. CMT-Bench is designed to probe robustness via three semantics-preserving dimensions: (i) extractive-cue ablation to separate extractive shortcuts from state tracking, (ii) temporal prefixing to test long-context stability, and (iii) entity-form perturbations (anonymization, out-of-distribution substitutions, role-entangling paraphrases) to assess sensitivity to surface variation. Across diverse long-context state-of-the-art LLMs, we find large drops without extractive summaries, monotonic degradation with input length, and consistent accuracy drop under entity-form changes. Complementary distributional tests confirm significant shifts in numeric error patterns, indicating drift in reasoning rather than mere noise. Our results show that current LLMs are brittle in dynamic Text-to-table generation, motivating robustness-first evaluation as a prerequisite for developing efficient and scalable approaches for this task.

Introduction

Structured data extraction—transforming unstructured text into tabular formats—is a critical application for Large Language Models (LLMs) in domains such as finance, sports, and clinical research. However, existing evaluation methods often fail to distinguish between genuine state-tracking reasoning and simple pattern matching. Many LLM-driven T2T systems are "brittle," achieving high performance by exploiting embedded summary cues rather than tracking state changes over time. CMT-Bench addresses this by providing a rigorous testbed using live cricket commentary, requiring models to maintain state across long contexts and multiple interacting entities.

The CMT-Bench Framework

CMT-Bench utilizes a dataset of over 6,500 commentary samples sourced from live ball-by-ball cricket coverage, including One Day International (ODI) and T20 matches. The task requires models to generate two evolving tables—a Batsman scorecard and a Bowler scorecard—based on the narrative flow of the commentary. The ground truth was programmatically generated and validated against official scorecards with a 99.7% match rate.

Example of the ball-by-ball commentary and the resulting structured table generation task in CMT-Bench.

Robustness Probes

To evaluate the depth of model reasoning, the authors introduced three diagnostic probes:

  1. Extractive-Cue Ablation: Performance is compared on text with summary cues (e.g., "Player X out for 50") versus text without them, forcing the model to calculate statistics ball-by-ball.
  2. Temporal Prefixing: Models are tested on varying "input horizons" (from 5 overs to a full 50-over match) to assess long-context stability.
  3. Entity-Form Perturbations: Sensitivity to surface variations is tested via anonymization (e.g., "Player1"), out-of-distribution (OOD) name substitutions (e.g., soccer players), and "Entity-Entanglement" (paraphrasing to break standard structures).

Key Results and Insights

The "Shortcut" Trap

Frontier models like Gemini 2.5 Flash and Llama-3.3 exhibit high precision as extractors but struggle as reasoners. When summary lines are removed, cell accuracy drops by an average of 42%, and row accuracy plummets by 87%. Interestingly, in standard commentary, batsman accuracy often appears to increase as the game progresses; however, this is an artifact of "inflation" caused by the increasing frequency of summary cues as more players are dismissed.

Long-Context Fragility and Reasoning Drift

Without the aid of summaries, models show a monotonic degradation in accuracy as the input length increases, struggling to maintain identity-consistent aggregation over thousands of tokens. "Entity Entanglement" proved to be the most disruptive perturbation, causing significant reasoning drift where models lose track of entity roles (e.g., confusing the bowler with the batsman), leading to a 15–20 point drop in accuracy.

Comparison of table-level accuracies across various LLMs, demonstrating performance degradation as the input horizon (Over Bucket) increases.

Model Performance

Gemini 2.5 Flash and Llama-70B were identified as the most robust models, yet even they showed significant distributional shifts under perturbation. Smaller models (in the 7B-8B range) were highly sensitive to name changes and phrasing, indicating a heavy reliance on lexical memorization.

Conclusion

The findings from CMT-Bench demonstrate that current LLMs are brittle in dynamic T2T generation tasks. While they excel at extraction, their ability to track state changes over long, evolving narratives is limited. Future work should focus on developing "rule-aware" decoding methods that enforce conservation constraints and expanding the benchmark to non-sports domains like financial filings or clinical trials.

Summary updated

Abstract

LLM Driven text-to-table (T2T) systems often rely on extensive prompt-engineering or iterative event extraction in code-parsable formats, which boosts scores but are computationally expensive and obscure how models actually reason over temporal evolving narratives to summarise key information. We present CMT-Bench, a diagnostic benchmark built from live cricket commentary that requires dynamic table generation across two evolving schemas under a dense, rule-governed policy. CMT-Bench is designed to probe robustness via three semantics-preserving dimensions: (i) extractive-cue ablation to separate extractive shortcuts from state tracking, (ii) temporal prefixing to test long-context stability, and (iii) entity-form perturbations (anonymization, outof-distribution substitutions, role-entangling paraphrases) to assess sensitivity to surface variation. Across diverse long-context stateof-the-art LLMs, we find large drops without extractive summaries, monotonic degradation with input length, and consistent accuracy drop under entity-form changes. Complementary distributional tests confirm significant shifts in numeric error patterns, indicating drift in reasoning rather than mere noise. Our results show that current LLMs are brittle in dynamic Textto-table generation, motivating robustness-first evaluation as a prerequisite for developing efficient and scalable approaches for this task.
BibTeX
@inproceedings{upadhyay2026cmt,
  author = {Ritam Upadhyay and Naman Ahuja and Rishabh Baral and Aparna Garimella and Vivek Gupta},
  title = {{CMT-Bench: Cricket Multi-Table Generation Benchmark for Probing Robustness in Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2072},
}
Figure