While Large Language Models (LLMs) have become remarkably adept at solving "coding interview" style problems, most current benchmarks—like HumanEval and MBPP—focus almost exclusively on Functional Programming (FP). In these tasks, a model simply writes a standalone function to transform an input into an output. However, real-world software engineering is built on Object-Oriented Programming (OOP).
A new study introduces the OOP benchmark, revealing a startling gap: models that excel at functional snippets often fail significantly when asked to design classes, manage state, or implement encapsulation.
The Functional Bias in LLM Evaluation
The "functional bias" in current benchmarks means we are likely overestimating how useful LLMs are for actual software development. In a production codebase, code isn't just a series of isolated functions; it’s a web of classes, objects, and inherited behaviors.
Existing metrics like $pass@k$ only check if the code produces the correct output for a given input. They don't verify if the model followed architectural constraints. For example, if a prompt asks for a private method to handle logic, a model might pass the unit tests by putting everything in a public method, technically "solving" the problem while failing the software design requirement.
Methodology: The OOP Benchmark and $pass@o$
To bridge this gap, the researchers introduced two major contributions:
- The OOP Benchmark: A collection of 431 Python programs across three difficulty levels (Simple, Moderate, and Difficult), manually rewritten from sources like LeetCode and GitHub into OOP structures.
- The $pass@o$ Metric: An "OOP-optimized" version of $pass@k$. To count as a success under $pass@o$, the generated code must pass the unit tests and contain the specific OOP structural elements (like the correct class name and private method prefixes) requested in the prompt.
The construction process, involving rigorous human rewriting and case design, is illustrated below:
Key Results and Insights
The performance drop-off from functional to object-oriented tasks is dramatic. As seen in the figure below, while WizardCoder-15b achieves nearly 60% on HumanEval, its score plummeted to a dismal 3.02% on the OOP benchmark.
The "Structural Illusion"
Many models generate code that works but isn't correct according to the design specifications. For instance, ChatGPT might solve a logic problem correctly but completely omit the required private function structure. This suggests that LLMs often suffer from a "structural illusion," where they prioritize functional correctness over architectural requirements.
Code-Specialized Models Lag in Design
Surprisingly, models specifically fine-tuned for code (like WizardCoder) often perform worse on OOP tasks than general-purpose models like ChatGPT. This suggests that code-specific training might be over-indexing on functional patterns found in competitive programming datasets, leading to a neglect of broader software engineering principles like encapsulation.
Limitations and Future Work
The current benchmark is limited to Python, and OOP looks very different in strictly typed languages like Java or C++. Furthermore, while the "Difficult" level includes inheritance, it doesn't yet touch on more complex patterns like multiple inheritance or advanced design patterns.
Developing OOP-specific prompting strategies is a clear next step, as the study found that standard Chain-of-Thought (CoT) prompting sometimes decreased performance by leading the model into structural errors. This paper serves as a wake-up call: being good at "coding puzzles" does not mean an LLM is ready to architect software.
Summary updated