Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in generating functionally correct code, their ability to navigate the complexities of software architecture remains a critical frontier. Functional correctness—passing unit tests—is only one facet of software engineering; maintainability and modularity are equally vital for long-term project health. This study investigates whether LLMs possess a foundational understanding of core design principles, specifically cohesion and coupling, or if they merely rely on mechanical pattern matching.
Methodology: A Hierarchical Approach
The researchers proposed a hierarchical evaluation framework inspired by Bloom’s Taxonomy to measure the "cognitive load" required for different design tasks. This framework evaluates models across three levels of increasing difficulty:
- Verification (Low Load): Confirming if a specific pair of classes exhibits tight coupling.
- Guided Generation (Medium Load): Identifying all classes coupled with a provided target class.
- Open-ended Generation (High Load): Independently discovering and partitioning all design flaws within a codebase.
The study focused on the DeepSeek-R1 model family (14B, 32B, and 70B parameters) to assess how scale and reasoning capabilities (via Chain-of-Thought) influence design performance.
Dataset Generation and Controlled Distortion
To create a robust benchmark, the authors utilized 210 mature open-source Java repositories. They programmatically introduced design flaws using the Spoon library to simulate real-world "spaghetti code."
These transformations involved injecting "noise" or "distractors"—mixing unrelated methods into classes (lowering cohesion) or creating artificial dependencies between modules (increasing coupling). The final dataset comprised 10,800 prompts per concept, balanced across task types and complexity levels.
Key Results and Insights
The evaluation revealed a significant performance gap between simple verification and complex architectural reasoning:
- The "Efficient Intern" Paradox: LLMs excel at following specific instructions in low-noise environments (e.g., the 70B model achieved an F1 of 0.899 for coupling in verification tasks). However, they lack the autonomy required for open-ended architectural discovery.
- Fragility of Coupling Reasoning: While models were relatively resilient when analyzing cohesion (intra-module), their understanding of coupling (inter-module) collapsed under noise. In open-ended scenarios with distractor code, the 70B model's F1 score for coupling plummeted by over 54%.
- Cognitive Shortcutting: Analysis of Chain-of-Thought (CoT) traces showed that models often produced shorter reasoning steps when faced with high-complexity coupling tasks, effectively "giving up" rather than exploring the full relationship space.
- Scale and Efficiency: Larger models (70B) were not only more accurate but also more concise, reaching correct conclusions with fewer reasoning steps, suggesting a deeper internalization of design principles.
Conclusion
The study concludes that while current LLMs like DeepSeek-R1 are becoming proficient "code writers," they are not yet "senior engineers." Their architectural reasoning is fragile and easily disrupted by the "noise" typical of large, complex codebases. Future work should explore the models' ability to not just identify but actively refactor these design flaws, moving toward the "Creating" level of Bloom’s Taxonomy.
Summary updated