MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

Jianbo Dai, Jianqiao Lu, Yunlong Feng, Dong Huang, Guangtao Zeng, Rongju Ruan, Ming Cheng, Haochen Tan, Zhijiang GuoSubmission to ICLR 2025 · Rejected

TL;DR

Existing code generation benchmarks like HumanEval and MBPP are increasingly saturated, plagued by data contamination, and limited to basic, formulaic tasks. To properly test frontier models, the authors introduce MHPP (Mostly Hard Python Problems)—a benchmark of 210 human-crafted, contamination-free problems spanning seven distinct cognitive and algorithmic challenges. Evaluated across 26 LLMs, pass rates plummet dramatically: even GPT-4o drops from over 90% on HumanEval to just 51.1% pass@1 on MHPP.

What Problem Does This Paper Address?

State-of-the-art models like GPT-4o now clear 90%+ pass rates on standard function-level coding benchmarks such as HumanEval. However, these benchmarks fail to reflect real-world programming competency due to three major flaws:

  1. Low difficulty and poor granularity: A large portion of HumanEval and MBPP questions consist of basic arithmetic or elementary standard library calls.
  2. Data contamination: A staggering 65.4% of MBPP test instances were detected as contaminated from publicly indexed websites (e.g., GeeksforGeeks).
  3. Question and test quality issues: In MBPP, nearly 19% of errors made by GPT-4 are attributable to missing constraints, missing parameter descriptions, or faulty unit tests rather than model limitations.

Because models often memorize these basic tasks, standard benchmarks can no longer effectively distinguish frontier model capabilities.

Methods

To construct a benchmark that genuinely tests model limits, the authors had a team of 12 CS researchers hand-craft problems categorized into seven specific reasoning challenges:

  • Distraction: Elaborate prompts (>200 words) containing extraneous background info, noise, or misleading descriptions.
  • Redefinition: Scenarios introducing counterfactual rules or custom operators that override pre-trained commonsense.
  • Shortcut: Problems that appear computationally expensive but allow concise, elegant solutions through lateral thinking, number theory, or game theory.
  • Commonsense: Implicit real-world physical, spatial, or temporal reasoning requirements not explicitly restated in the prompt.
  • Cornercase: Functions requiring explicit handling of subtle boundary conditions (e.g., degenerate geometries).
  • Complexity: Multi-step problems requiring three or more reasoning hops, dynamic programming, or rigorous planning.
  • Codesense: Requirements demanding precise API parameters and advanced library usage (e.g., numpy, re).

To prevent leakage, the dataset underwent strict contamination filtering (achieving 0% detection), and test suites are withheld behind an evaluation API rather than released in raw plain text.

Datasets

  • MHPP (Mostly Hard Python Problems): The newly introduced benchmark. Contains 210 problems (30 per challenge category). Compared to HumanEval and MBPP, MHPP features longer problem descriptions (167.6 words vs. 23.0 in HumanEval), longer reference implementations (14.9 lines vs. 6.3), and significantly denser unit test suites (14.0 test cases per problem vs. 7.2 in HumanEval and 3.0 in MBPP).
  • Cross-Language Subset: 140 problems translated into Java and C++ to test language generalization.
  • Prior Benchmarks Analyzed: HumanEval and MBPP.

Key Insights and Ideas

  • The "Generalization Gap" Widens: While open-source models (such as Llama 3.1 405B and DeepSeek-V2.5) closely trail GPT-4o on HumanEval, the gap reopens on MHPP. GPT-4o maintains a commanding lead over all contenders.
  • Where LLMs Fall Apart:
  • Algorithmic Shortcuts: Models struggle the most with Shortcut and Complex problems. On Shortcut tasks, even GPT-4-Turbo fails ~60% of the time, often defaulting to brute-force loops that time out.
  • Spatial Commonsense: Models repeatedly fail to understand spatial orientation (e.g., simulating commuter collisions on a line) when rules aren't completely spoon-fed.
  • Constraint Overload: Under complex instructions, models frequently lose track of zero-based indexing or misapply intermediate constraints midway through the code.
  • Steep Cross-Language Drop: Testing GPT-4 on the Java and C++ translations revealed a severe performance drop (53.6% pass@1 in Python vs. 25.0% in Java and 27.9% in C++), indicating strong Python-specific training bias.

Key Results

Evaluations across 26 LLMs (evaluated via pass@1 and pass@5 with sampling at temperature 0.7, as well as greedy decoding) demonstrate that MHPP is substantially harder than legacy benchmarks:

Model HumanEval pass@1 MHPP pass@1 MHPP pass@5 MHPP Greedy
GPT-4o (2024-05-13) ~91.0% 51.1% 64.4% 51.4%
GPT-4-Turbo (2024-04-09) ~87% 45.1% 58.7% 43.8%
GPT-4o-Mini — 44.2% 56.8% 42.4%
DeepSeek-V2.5 High 80s 42.0% 51.3% 40.0%
Llama 3.1 405B (Greedy) — — — 39.5%
Llama 3.1 70B (Greedy) — — — 33.8%
GPT-3.5-Turbo-0125 ~75% 29.6% 46.0% 28.1%
Llama 3.1 8B ~65% 7.1% 18.4% 17.1%

Limitations and Future Work

  • Dataset Scale: Because each problem was manually authored and verified by human experts, MHPP is relatively small (210 problems), which limits granular statistical analysis across sub-domains.
  • Scope: MHPP remains strictly a function-level benchmark; it does not evaluate repository-level reasoning, cross-file navigation, or multi-turn conversational debugging (like SWE-bench).
  • Future Work: Developing robust methods to scale human-quality dataset curation, extending MHPP's challenge taxonomy to multi-file software engineering tasks, and leveraging execution-based feedback to help models overcome edge-case blind spots.

Summary updated

Abstract

Recent advancements in large language models (LLMs) have greatly improved code generation, specifically at the function level. For instance, GPT-4o has achieved a 91.0\% pass rate on HumanEval. However, this draws into question the adequacy of existing benchmarks in thoroughly assessing function-level code generation capabilities. Our study analyzed two common benchmarks, HumanEval and MBPP, and found that these might not thoroughly evaluate LLMs' code generation capacities due to limitations in quality, difficulty, and granularity. To resolve this, we introduce the Mostly Hard Python Problems (MHPP) dataset, consisting of 210 unique human-curated problems. By focusing on the combination of natural language and code reasoning, MHPP gauges LLMs' abilities to comprehend specifications and restrictions, engage in multi-step reasoning, and apply coding knowledge effectively. Initial evaluations of 26 LLMs using MHPP showed many high-performing models on HumanEval failed to achieve similar success on MHPP. Moreover, MHPP highlighted various previously undiscovered limitations within various LLMs, leading us to believe that it could pave the way for a better understanding of LLMs' capabilities and limitations.
BibTeX
@misc{dai2024mhpp,
  author = {Jianbo Dai and Jianqiao Lu and Yunlong Feng and Dong HUANG and Guangtao Zeng and Rongju Ruan and Ming Cheng and Haochen Tan and Zhijiang Guo},
  title = {{MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation}},
  year = {2024},
  eprint = {2405.11430},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.11430},
}
Figure