Hierarchical Evaluation of Software Design Capabilities of Large Language Models of Code

Mootez Saad, Boqi Chen, José Antonio Hernández López, Dániel Varró, Tushar SharmaarXiv 2025 · DOI · arXiv

Introduction

While Large Language Models (LLMs) have demonstrated remarkable proficiency in generating functionally correct code, their ability to navigate the complexities of software architecture remains a critical frontier. Functional correctness—passing unit tests—is only one facet of software engineering; maintainability and modularity are equally vital for long-term project health. This study investigates whether LLMs possess a foundational understanding of core design principles, specifically cohesion and coupling, or if they merely rely on mechanical pattern matching.

Methodology: A Hierarchical Approach

The researchers proposed a hierarchical evaluation framework inspired by Bloom’s Taxonomy to measure the "cognitive load" required for different design tasks. This framework evaluates models across three levels of increasing difficulty:

  1. Verification (Low Load): Confirming if a specific pair of classes exhibits tight coupling.
  2. Guided Generation (Medium Load): Identifying all classes coupled with a provided target class.
  3. Open-ended Generation (High Load): Independently discovering and partitioning all design flaws within a codebase.
Overview of the methodology pipeline, illustrating the hierarchical prompting levels and the analysis flow.

The study focused on the DeepSeek-R1 model family (14B, 32B, and 70B parameters) to assess how scale and reasoning capabilities (via Chain-of-Thought) influence design performance.

Dataset Generation and Controlled Distortion

To create a robust benchmark, the authors utilized 210 mature open-source Java repositories. They programmatically introduced design flaws using the Spoon library to simulate real-world "spaghetti code."

Illustration of the code transformation process where responsibilities from source classes are injected into a target class to simulate design flaws.

These transformations involved injecting "noise" or "distractors"—mixing unrelated methods into classes (lowering cohesion) or creating artificial dependencies between modules (increasing coupling). The final dataset comprised 10,800 prompts per concept, balanced across task types and complexity levels.

Key Results and Insights

The evaluation revealed a significant performance gap between simple verification and complex architectural reasoning:

  • The "Efficient Intern" Paradox: LLMs excel at following specific instructions in low-noise environments (e.g., the 70B model achieved an F1 of 0.899 for coupling in verification tasks). However, they lack the autonomy required for open-ended architectural discovery.
  • Fragility of Coupling Reasoning: While models were relatively resilient when analyzing cohesion (intra-module), their understanding of coupling (inter-module) collapsed under noise. In open-ended scenarios with distractor code, the 70B model's F1 score for coupling plummeted by over 54%.
  • Cognitive Shortcutting: Analysis of Chain-of-Thought (CoT) traces showed that models often produced shorter reasoning steps when faced with high-complexity coupling tasks, effectively "giving up" rather than exploring the full relationship space.
  • Scale and Efficiency: Larger models (70B) were not only more accurate but also more concise, reaching correct conclusions with fewer reasoning steps, suggesting a deeper internalization of design principles.

Conclusion

The study concludes that while current LLMs like DeepSeek-R1 are becoming proficient "code writers," they are not yet "senior engineers." Their architectural reasoning is fragile and easily disrupted by the "noise" typical of large, complex codebases. Future work should explore the models' ability to not just identify but actively refactor these design flaws, moving toward the "Creating" level of Bloom’s Taxonomy.

Summary updated

BibTeX
@misc{saad2025hierarchical,
  author = {Mootez Saad and Boqi Chen and José Antonio Hernández López and Dániel Varró and Tushar Sharma},
  title = {{Hierarchical Evaluation of Software Design Capabilities of Large Language Models of Code}},
  year = {2025},
  eprint = {2511.20933},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.20933},
}
Figure