Abstract
Recent frontier-level LLMs have saturated many previously difficult benchmarks, leaving little room for further differentiation. This progress highlights the need for challenging benchmarks that provide objective verification. In this paper, we introduce MCBench, a benchmark designed to evaluate whether LLMs can execute string-matching NLP metrics by strictly following step-by-step instructions. Unlike prior benchmarks that depend on subjective judgments or general reasoning, MCBench offers an objective, deterministic and code-verifiable evaluation. This setup allows us to systematically test whether LLMs can maintain accurate step-by-step execution, including instruction adherence, numerical computation, and long-range consistency in handling intermediate results. To ensure objective evaluation of these abilities, we provide a parallel reference code that can evaluate the accuracy of LLM output. We provide three evaluative metrics and three benchmark variants designed to measure the detailed instruction understanding capability of LLMs. Our analyses show that MCBench serves as an effective and objective tool for evaluating the capabilities of cutting-edge LLMs.
TL;DR
MCBench is a new, objective benchmark that challenges Large Language Models (LLMs) to perform a task they usually outsource: manually calculating complex NLP string-matching metrics (like BLEU or ROUGE) following a strict, multi-step rubric. The results are a wake-up call—even frontier models like GPT-4o struggle with the long-range consistency required, achieving only about 41% accuracy.
The Problem: Benchmark Saturation and Subjectivity
We are currently witnessing a "benchmark saturation" crisis in AI. Popular evaluations like MATH and IFEval, once considered formidable, are being "solved" by the latest models with near-perfect scores. Furthermore, many modern benchmarks rely on "LLM-as-a-judge," which introduces human-like subjectivity and bias into the evaluation process.
The authors of MCBench argue that we need tougher, deterministic, and code-verifiable benchmarks. They focus on the gap between a model "knowing" a concept and being able to "execute" it. While an LLM can easily define the BLEU score, can it actually perform the seven distinct steps of tokenization, n-gram counting, and brevity penalty calculation on a specific pair of sentences without making a mistake? MCBench tests this "instruction-following" endurance.
The MCBench Framework
The core of MCBench is a set of rubrics for 33 different string-matching metrics. Each rubric is broken down into three components: Requirements (natural language description), Example (concrete walkthrough), and Code (Python implementation).
To evaluate the models, the researchers developed a parallel reference code. This allows them to programmatically verify not just the final answer, but also the intermediate steps. They use three primary evaluative metrics:
- Final Accuracy (FA): Did the model get the right number (within a 5% error bound)?
- Format Following (FF): Did the model adhere to the specific output structure requested?
- Following Depth (FD): How many intermediate steps did the model complete correctly before failing?
To prevent models from simply "remembering" the formulas from their training data, the researchers replaced specific metric names with the neutral label "NLP score" during testing.
Dataset Composition
The MCBench dataset includes 33 metrics ranging from common ones like BLEU and ROUGE to more obscure ones like the Tversky index and Lee Distance. These are tested across 1,320 instances in four categories:
- Plain: Standard English sentences.
- Dzongkha: A low-resource language to test tokenizer-parity issues.
- Emoji: Testing extreme modality shifts.
- Harmful: Sentences containing jailbreak prompts or harmful content to see if the model "overreacts" and refuses the calculation task.
Key Insights
- The Execution Gap: There is a massive difference between a model's internal knowledge and its ability to maintain consistency over a long sequence of operations. Most failures occurred because models lost track of intermediate values.
- Reasoning vs. Formatting: "Reasoning" models like QwQ-32B achieved the highest accuracy but were the worst at following formatting instructions. This suggests that optimizing for deep "thinking" may reduce compliance with strict structural constraints.
- Information Overload: Counter-intuitively, providing more information (Requirements + Example + Code) often acted as "noise," leading to lower performance compared to just providing the Requirements.
- Modality Sensitivity: Performance plummeted on Emoji and Harmful categories, revealing that string matching becomes significantly harder when input text deviates from standard distributions.
Key Results
Frontier models found the benchmark surprisingly difficult. GPT-4o achieved a final accuracy of only 40.98%, while smaller models like Llama 3.1-8B scored around 16.74%.
As shown in the metric-by-metric analysis, specialized reasoning models like QwQ-32B show significant gains on specific metrics (e.g., Jaccard Distance or Bag distance) but still struggle with others like Character Error Rate. The study also confirmed that using the term "NLP score" had a negligible effect, proving the benchmark tests instruction following rather than memory retrieval.
Limitations and Future Work
MCBench is currently limited to deterministic string-matching metrics and excludes probabilistic measures like BERTScore. Additionally, the benchmark intentionally excludes external tools to isolate internal reasoning. Future work aims to expand the benchmark to include tool implementation and a broader variety of NLP primitives.
Summary updated