OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models

Shuai Wang, Liang Ding, Li Shen, Yong Luo, Bo Du, Dacheng Tao

While Large Language Models (LLMs) have become remarkably adept at solving "coding interview" style problems, most current benchmarks—like HumanEval and MBPP—focus almost exclusively on Functional Programming (FP). In these tasks, a model simply writes a standalone function to transform an input into an output. However, real-world software engineering is built on Object-Oriented Programming (OOP).

A new study introduces the OOP benchmark, revealing a startling gap: models that excel at functional snippets often fail significantly when asked to design classes, manage state, or implement encapsulation.

The Functional Bias in LLM Evaluation

The "functional bias" in current benchmarks means we are likely overestimating how useful LLMs are for actual software development. In a production codebase, code isn't just a series of isolated functions; it’s a web of classes, objects, and inherited behaviors.

Existing metrics like $pass@k$ only check if the code produces the correct output for a given input. They don't verify if the model followed architectural constraints. For example, if a prompt asks for a private method to handle logic, a model might pass the unit tests by putting everything in a public method, technically "solving" the problem while failing the software design requirement.

Methodology: The OOP Benchmark and $pass@o$

To bridge this gap, the researchers introduced two major contributions:

  1. The OOP Benchmark: A collection of 431 Python programs across three difficulty levels (Simple, Moderate, and Difficult), manually rewritten from sources like LeetCode and GitHub into OOP structures.
  2. The $pass@o$ Metric: An "OOP-optimized" version of $pass@k$. To count as a success under $pass@o$, the generated code must pass the unit tests and contain the specific OOP structural elements (like the correct class name and private method prefixes) requested in the prompt.

The construction process, involving rigorous human rewriting and case design, is illustrated below:

The construction process of the OOP benchmark, moving from raw source questions to rewritten prompts and test cases.

Key Results and Insights

The performance drop-off from functional to object-oriented tasks is dramatic. As seen in the figure below, while WizardCoder-15b achieves nearly 60% on HumanEval, its score plummeted to a dismal 3.02% on the OOP benchmark.

Comparison of scores on HumanEval/MBPP (Functional) vs. the new OOP benchmark. Note the massive performance gap for all models.

The "Structural Illusion"

Many models generate code that works but isn't correct according to the design specifications. For instance, ChatGPT might solve a logic problem correctly but completely omit the required private function structure. This suggests that LLMs often suffer from a "structural illusion," where they prioritize functional correctness over architectural requirements.

Code-Specialized Models Lag in Design

Surprisingly, models specifically fine-tuned for code (like WizardCoder) often perform worse on OOP tasks than general-purpose models like ChatGPT. This suggests that code-specific training might be over-indexing on functional patterns found in competitive programming datasets, leading to a neglect of broader software engineering principles like encapsulation.

Limitations and Future Work

The current benchmark is limited to Python, and OOP looks very different in strictly typed languages like Java or C++. Furthermore, while the "Difficult" level includes inheritance, it doesn't yet touch on more complex patterns like multiple inheritance or advanced design patterns.

Developing OOP-specific prompting strategies is a clear next step, as the study found that standard Chain-of-Thought (CoT) prompting sometimes decreased performance by leading the model into structural errors. This paper serves as a wake-up call: being good at "coding puzzles" does not mean an LLM is ready to architect software.

Summary updated

BibTeX
@inproceedings{wang2024oop,
  author = {Shuai Wang and Liang Ding and Li Shen and Yong Luo and Bo Du and Dacheng Tao},
  title = {{OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=M2oJVmFs6b},
}
Figure