Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models

Hyeonseok Moon, Seongtae Hong, Jaehyung Seo, Heuiseok Lim

Abstract

Recent frontier-level LLMs have saturated many previously difficult benchmarks, leaving little room for further differentiation. This progress highlights the need for challenging benchmarks that provide objective verification. In this paper, we introduce MCBench, a benchmark designed to evaluate whether LLMs can execute string-matching NLP metrics by strictly following step-by-step instructions. Unlike prior benchmarks that depend on subjective judgments or general reasoning, MCBench offers an objective, deterministic and code-verifiable evaluation. This setup allows us to systematically test whether LLMs can maintain accurate step-by-step execution, including instruction adherence, numerical computation, and long-range consistency in handling intermediate results. To ensure objective evaluation of these abilities, we provide a parallel reference code that can evaluate the accuracy of LLM output. We provide three evaluative metrics and three benchmark variants designed to measure the detailed instruction understanding capability of LLMs. Our analyses show that MCBench serves as an effective and objective tool for evaluating the capabilities of cutting-edge LLMs.

Introduction: Beyond Benchmark Saturation

As Large Language Models (LLMs) continue to evolve, standard benchmarks like MATH and IFEval are reaching saturation points where frontier models achieve near-perfect scores. This makes it increasingly difficult to differentiate between top-tier models. Furthermore, many existing evaluations rely on "LLM-as-a-judge" frameworks or human review, both of which introduce subjectivity and potential bias.

MCBench addresses this by focusing on Complex Instruction Following, Mathematical Reasoning, and Long-Range Consistency. The benchmark requires models to manually calculate classic NLP string-matching metrics (such as BLEU, ROUGE, or Hamming Distance) by following a strict, multi-step rubric. This task is inherently deterministic, allowing for a purely objective, code-verifiable assessment of a model's ability to execute complex procedures rather than just reciting memorized knowledge.

Methodology and Framework

The core of MCBench is the transformation of NLP evaluation metrics into the evaluation task itself. The framework consists of three primary components designed to ensure a rigorous and objective testing environment.

The Rubric Structure

For each of the 33 included metrics, a detailed rubric is provided. This rubric is divided into three distinct parts:

  1. Requirements: Natural language steps describing the calculation process.
  2. Examples: A step-by-step walkthrough of a sample calculation.
  3. Code: A Python implementation of the steps to provide unambiguous logic.

To prevent models from relying on memorized definitions of famous metrics, the actual names (e.g., "BLEU") are masked with the neutral label "NLP score" during testing.

Objective Verification

Because the tasks are deterministic, the authors developed "Parallel Reference Code" for every metric. This allows for the automatic verification of not just the final numerical answer, but every intermediate step the LLM produces. This granular verification is essential for identifying exactly where a model's reasoning or execution fails.

Overview of the MCBench framework, illustrating the rubric structure and verification process.

Benchmark Design and Evaluation Metrics

MCBench utilizes 33 different string-matching metrics, ranging from simple Levenshtein distance to complex scores like CIDEr and NIST. These metrics are applied to four "Candidate Set" categories to test robustness across different linguistic and symbolic domains:

  • Plain: Standard English sentence pairs.
  • Dzongkha: A low-resource language used to test tokenizer-parity and linguistic robustness.
  • Emoji: Testing modality shifts where the text consists entirely of non-alphanumeric symbols.
  • Harmful: Sentence pairs involving "jailbreak" prompts to determine if safety filters interfere with neutral calculation instructions.

Performance is measured using three primary metrics: Final Accuracy (FA), which checks if the final result is within a 5% error bound; Format Following (FF), which assesses adherence to the output template; and Following Depth (FD), which measures how many intermediate steps were completed correctly before the first error occurred.

Key Results and Analysis

The results from MCBench reveal a significant "Knowing vs. Doing" gap in current LLMs. While these models are trained on the definitions of these metrics, the manual execution of the multi-step calculation remains a major hurdle.

Comparative performance of various LLMs across different rubric configurations and candidate sets.

Frontier Model Performance

Even the most advanced models struggle with the benchmark. GPT-4o achieved a Final Accuracy of only 40.98%. The top performer, QwQ-32B (a reasoning-distilled model), reached an average accuracy of 48.03%. However, a notable trade-off was observed: while reasoning-oriented models like QwQ-32B achieved higher accuracy, they were significantly worse at following formatting instructions (FF score of 54.02%) compared to standard models. This suggests that intensive reasoning may come at the cost of strict adherence to output templates.

The Tokenization Bottleneck

Analysis of the Following Depth (FD) metric shows that errors frequently occur at the very first step: tokenization. This is particularly evident in the Emoji and Dzongkha sets. If the initial tokenization is incorrect, the subsequent mathematical operations inevitably fail, leading to a total collapse of the calculation chain.

Information Overload

Interestingly, providing more information in the rubric did not always lead to better performance. For many models, clear natural language requirements were more effective than the addition of complex code snippets, which occasionally confused the models rather than clarifying the task.

Conclusion and Future Work

MCBench demonstrates that complex instruction following remains a significant challenge even for frontier LLMs. The benchmark successfully isolates execution capabilities from memorized knowledge through its deterministic, code-verifiable design.

Future iterations of MCBench may expand beyond string-matching to include probabilistic NLP primitives or graph-based scores. Additionally, the authors suggest incorporating "tool implementation" tasks, where models are evaluated on their ability to write and execute their own verification code to solve these complex instructions, rather than relying solely on internal reasoning.

Summary updated

Abstract

Recent frontier-level LLMs have saturated many previously difficult benchmarks, leaving little room for further differentiation.This progress highlights the need for challenging benchmarks that provide objective verification.In this paper, we introduce MCBench, a benchmark designed to evaluate whether LLMs can execute string-matching NLP metrics by strictly following step-by-step instructions.Unlike prior benchmarks that depend on subjective judgments or general reasoning, MCBench offers an objective, deterministic and codeverifiable evaluation.This setup allows us to systematically test whether LLMs can maintain accurate step-by-step execution, including instruction adherence, numerical computation, and long-range consistency in handling intermediate results.To ensure objective evaluation of these abilities, we provide a parallel reference code that can evaluate the accuracy of LLM output.We provide three evaluative metrics and three benchmark variants designed to measure the detailed instruction understanding capability of LLMs.Our analyses show that MCBench serves as an effective and objective tool for evaluating the capabilities of cuttingedge LLMs.
BibTeX
@misc{moon2025metric,
  author = {Hyeonseok Moon and Seongtae Hong and Jaehyung Seo and Heuiseok Lim},
  title = {{Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models}},
  year = {2025},
  eprint = {2510.07892},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.07892},
}
Figure