TL;DR
As large language models (LLMs) achieve near-perfect scores on classic coding benchmarks, it becomes harder to tell if they are truly mastering programming or just memorizing templates. To address this, the authors introduce MHPP (Mostly Hard Python Problems), a human-curated dataset of 210 code generation tasks designed to test advanced reasoning and edge cases. Evaluation of 26 leading LLMs reveals a massive performance drop compared to traditional benchmarks, proving that even the best models still struggle when forced to think outside the box.
What Problem Does This Paper Address?
State-of-the-art models like GPT-4o boast a staggering 91.0% pass rate on HumanEval, leading many to wonder if function-level code generation is a "solved" problem. However, the authors argue that existing benchmarks suffer from three major flaws:
- Data Contamination: A leakage analysis revealed that up to 65.4% of the MBPP dataset is contaminated, meaning models might simply be memorizing solutions from their training data.
- Lack of Difficulty: Most problems only require basic arithmetic or standard library calls.
- Imbalanced Granularity: The problems fail to evaluate specific, fine-grained programming capabilities like handling ambiguous constraints or edge cases.
To bridge this gap, the paper aims to establish a more rigorous, contamination-free benchmark that forces LLMs to demonstrate genuine natural language comprehension and multi-step algorithmic reasoning.
Methods
The authors designed the MHPP benchmark by first analyzing the exact types of errors prominent models make on HumanEval.
Based on this error analysis, they hired a team of 12 computer science experts (Masters and PhDs) to manually curate problems across 7 distinct challenge categories:
- Distraction: Lengthy descriptions containing redundant or misleading information.
- Redefinition: Introducing counterfactual rules or entirely new concepts that contradict standard programming assumptions.
- Shortcut: Problems requiring lateral thinking, math shortcuts, or game theory rather than brute-force coding.
- Commonsense: Relying on implicit real-world knowledge (e.g., spatial or physical concepts) not explicitly detailed in the prompt.
- Cornercase: Demanding strict adherence to implicit boundary conditions.
- Complexity: Requiring multiple operational steps or reasoning hops.
- Codesense: Requiring deep proficiency in multiple internal or external Python packages (like NumPy or
re).
To ensure long-term benchmark integrity, the authors built a secure evaluation pipeline. Instead of open-sourcing the full dataset text where web crawlers can scrape it, researchers must submit model outputs via an API to get a performance report.
Datasets
The paper evaluates models using:
- HumanEval & MBPP: Used for baseline error analysis and correlation tracking.
- MHPP (Mostly Hard Python Problems): The newly introduced dataset consisting of 210 unique, human-curated problems (30 per challenge category).
Compared to HumanEval, MHPP problems are significantly more verbose (averaging 167.6 words vs. 23.0) and feature a much higher density of test cases (14.0 vs. 7.2 per problem) to prevent false positives.
Key Insights and Ideas
- The Scaling Illusion: While scaling up model size yields massive, predictable gains on HumanEval, the benefits are far less pronounced on MHPP for certain model families (like Gemma2 and Mixtral). This strongly implies that these models are overfitting to the style of older benchmarks rather than developing robust reasoning capabilities.
- Algorithmic Blindspots: Across the board, LLMs perform worst on "Shortcut" and "Complex" tasks. Models struggle heavily with abstract mathematical optimization and multi-step logic planning.
- Spatial and Physical Failures: In the "Commonsense" category, models frequently fail at basic real-world concepts. For instance, in a problem simulating a car commute, models struggled to grasp basic spatial awareness, calculating encounters for cars moving away from each other rather than toward each other.
Key Results
- The Performance Drop: GPT-4o's performance plummets from its 91.0% high on HumanEval down to 51.1% pass@1 on MHPP.
- Open-Source Champions: DeepSeek-V2.5 emerged as the strongest open-source model, achieving a 42.0% pass@1 score, notably outperforming closed-source alternatives like GPT-3.5-Turbo (29.6%).
- Cross-Language Friction: When translating a subset of MHPP into Java and C++, GPT-4's performance dropped sharply (from 53.6% in Python to 25.0% in Java and 27.9% in C++), indicating that advanced algorithmic reasoning becomes even harder for LLMs when decoupled from their most heavily trained language.
Limitations and Future Work
The primary limitation acknowledged by the authors is dataset scale. Because MHPP relies entirely on strict human curation and multi-stage meta-reviewing, it contains only 210 problems. Additionally, the benchmark focuses strictly on function-level code generation, which does not fully capture the broader context of repository-level software engineering (e.g., managing multi-file dependencies or system architectures).
Future work will focus on developing high-quality scaling strategies for human-curated datasets and expanding the benchmark's scope to encompass more complex, multi-file software development environments.
Summary updated