MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

Jianbo Dai, Jianqiao Lu, Yunlong Feng, Guangtao Zeng, Rongju Ruan, Ming Cheng, Dong Huang, Haochen Tan, Zhijiang Guo

TL;DR

As large language models (LLMs) achieve near-perfect scores on classic coding benchmarks, it becomes harder to tell if they are truly mastering programming or just memorizing templates. To address this, the authors introduce MHPP (Mostly Hard Python Problems), a human-curated dataset of 210 code generation tasks designed to test advanced reasoning and edge cases. Evaluation of 26 leading LLMs reveals a massive performance drop compared to traditional benchmarks, proving that even the best models still struggle when forced to think outside the box.

What Problem Does This Paper Address?

State-of-the-art models like GPT-4o boast a staggering 91.0% pass rate on HumanEval, leading many to wonder if function-level code generation is a "solved" problem. However, the authors argue that existing benchmarks suffer from three major flaws:

  1. Data Contamination: A leakage analysis revealed that up to 65.4% of the MBPP dataset is contaminated, meaning models might simply be memorizing solutions from their training data.
  2. Lack of Difficulty: Most problems only require basic arithmetic or standard library calls.
  3. Imbalanced Granularity: The problems fail to evaluate specific, fine-grained programming capabilities like handling ambiguous constraints or edge cases.

To bridge this gap, the paper aims to establish a more rigorous, contamination-free benchmark that forces LLMs to demonstrate genuine natural language comprehension and multi-step algorithmic reasoning.

Methods

The authors designed the MHPP benchmark by first analyzing the exact types of errors prominent models make on HumanEval.

Figure 2: Distribution of error types of GPT- 3.5, GPT-4 and DeepSeek-Instruct 33B on Hu- manEval. Models tend to make mistakes on prob- lems of the Complex type, while they are good at Basic, Distraction, and Codesense types.

Based on this error analysis, they hired a team of 12 computer science experts (Masters and PhDs) to manually curate problems across 7 distinct challenge categories:

  • Distraction: Lengthy descriptions containing redundant or misleading information.
  • Redefinition: Introducing counterfactual rules or entirely new concepts that contradict standard programming assumptions.
  • Shortcut: Problems requiring lateral thinking, math shortcuts, or game theory rather than brute-force coding.
  • Commonsense: Relying on implicit real-world knowledge (e.g., spatial or physical concepts) not explicitly detailed in the prompt.
  • Cornercase: Demanding strict adherence to implicit boundary conditions.
  • Complexity: Requiring multiple operational steps or reasoning hops.
  • Codesense: Requiring deep proficiency in multiple internal or external Python packages (like NumPy or re).

To ensure long-term benchmark integrity, the authors built a secure evaluation pipeline. Instead of open-sourcing the full dataset text where web crawlers can scrape it, researchers must submit model outputs via an API to get a performance report.

Datasets

The paper evaluates models using:

  • HumanEval & MBPP: Used for baseline error analysis and correlation tracking.
  • MHPP (Mostly Hard Python Problems): The newly introduced dataset consisting of 210 unique, human-curated problems (30 per challenge category).

Compared to HumanEval, MHPP problems are significantly more verbose (averaging 167.6 words vs. 23.0) and feature a much higher density of test cases (14.0 vs. 7.2 per problem) to prevent false positives.

Figure 1: A concise example from MHPP. The function is defined (1), documented with a de- scription in its docstring (2), and is accompanied by an input example (3). A canonical answer is presented (4), and the function’s correctness is ensured through an assertion test (5).

Key Insights and Ideas

  • The Scaling Illusion: While scaling up model size yields massive, predictable gains on HumanEval, the benefits are far less pronounced on MHPP for certain model families (like Gemma2 and Mixtral). This strongly implies that these models are overfitting to the style of older benchmarks rather than developing robust reasoning capabilities.
  • Algorithmic Blindspots: Across the board, LLMs perform worst on "Shortcut" and "Complex" tasks. Models struggle heavily with abstract mathematical optimization and multi-step logic planning.
  • Spatial and Physical Failures: In the "Commonsense" category, models frequently fail at basic real-world concepts. For instance, in a problem simulating a car commute, models struggled to grasp basic spatial awareness, calculating encounters for cars moving away from each other rather than toward each other.

Key Results

  • The Performance Drop: GPT-4o's performance plummets from its 91.0% high on HumanEval down to 51.1% pass@1 on MHPP.
  • Open-Source Champions: DeepSeek-V2.5 emerged as the strongest open-source model, achieving a 42.0% pass@1 score, notably outperforming closed-source alternatives like GPT-3.5-Turbo (29.6%).
  • Cross-Language Friction: When translating a subset of MHPP into Java and C++, GPT-4's performance dropped sharply (from 53.6% in Python to 25.0% in Java and 27.9% in C++), indicating that advanced algorithmic reasoning becomes even harder for LLMs when decoupled from their most heavily trained language.

Limitations and Future Work

The primary limitation acknowledged by the authors is dataset scale. Because MHPP relies entirely on strict human curation and multi-stage meta-reviewing, it contains only 210 problems. Additionally, the benchmark focuses strictly on function-level code generation, which does not fully capture the broader context of repository-level software engineering (e.g., managing multi-file dependencies or system architectures).

Future work will focus on developing high-quality scaling strategies for human-curated datasets and expanding the benchmark's scope to encompass more complex, multi-file software development environments.

Summary updated

Abstract

Recent advancements in large language models (LLMs) have greatly improved code generation, specifically at the function level. For instance, GPT-4o has achieved a 91.0\% pass rate on HumanEval. However, this draws into question the adequacy of existing benchmarks in thoroughly assessing function-level code generation capabilities. Our study analyzed two common benchmarks, HumanEval and MBPP, and found that these might not thoroughly evaluate LLMs' code generation capacities due to limitations in quality, difficulty, and granularity. To resolve this, we introduce the Mostly Hard Python Problems (MHPP) dataset, consisting of 210 unique human-curated problems. By focusing on the combination of natural language and code reasoning, MHPP gauges LLMs' abilities to comprehend specifications and restrictions, engage in multi-step reasoning, and apply coding knowledge effectively. Initial evaluations of 26 LLMs using MHPP showed many high-performing models on HumanEval failed to achieve similar success on MHPP. Moreover, MHPP highlighted various previously undiscovered limitations within various LLMs, leading us to believe that it could pave the way for a better understanding of LLMs' capabilities and limitations. MHPP, evaluation pipeline, and leaderboard can be found in https://github.com/SparksofAGI/MHPP.
BibTeX
@misc{dai2024mhpp,
  author = {Jianbo Dai and Jianqiao Lu and Yunlong Feng and Guangtao Zeng and Rongju Ruan and Ming Cheng and Dong Huang and Haochen Tan and Zhijiang Guo},
  title = {{MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation}},
  year = {2024},
  eprint = {2405.11430},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.11430},
}
Figure