Revisiting the Impact of Pursuing Modularity for Code Generation

Deokyeong Kang, Ki Jung Seo, Taeuk KimEMNLP 2024 · DOI

Introduction

In traditional software engineering, modularity—the practice of decomposing a program into small, independent functions—is considered a "gold standard." It enhances readability, reusability, and maintainability for human developers by reducing cognitive load. Consequently, it has been widely assumed that providing modular code examples to Large Language Models (LLMs) would similarly improve their code generation performance.

However, LLMs do not process information like humans. They do not suffer from the same cognitive constraints that make a lengthy, monolithic function difficult to follow. This paper investigates whether the pursuit of modularity actually benefits LLMs or if it is merely a human-centric preference that does not translate to machine learning performance.

Comparison of Modular vs. Non-Modular code structures.

Quantifying Modularity: The MOS Metric

To rigorously test the impact of modularity, the researchers introduced a quantitative measure called the Modularity Score (MOS). This metric is built upon Cyclomatic Complexity (CC), which counts the number of linearly independent paths through a program's source code.

The MOS calculation involves determining an "ideal" number of modules ($m^*$) based on a complexity threshold and comparing it to the actual number of modules ($n$) present in the code. By using MOS, the authors categorized code into four types: existing Modular Code (MC), Singular Code (SC), Transformed Modular Code (TMC), and Transformed Singular Code (TSC). This allowed them to isolate the structural impact of modularity from other factors like logic simplification.

The workflow for calculating the Modularity Score (MOS) using Control Flow Graphs (CFG) and Cyclomatic Complexity (CC).

Key Insights: Modularity is for Humans, Not Models

The study's findings challenge the conventional wisdom of modularity in the context of AI:

  • The "Cleaning Effect" Illusion: Previous studies suggesting modularity helps often compared "messy" original code to "clean" modularized code. This research shows that if you take clean modular code and "flatten" it back into a single block (TSC), the performance remains virtually identical. The improvement observed in prior work likely came from the LLM "cleaning" the logic during modularization, not the modular structure itself.
  • Performance Disparity: In many tests, Singular Code (SC) outperformed Modular Code (MC). For instance, on the APPS dataset, DeepSeekCoder 6.7B achieved a pass@1 of 10.74 with singular code compared to 9.39 with modular code. In fine-tuning on CodeContests, the gap was even wider (8.73 for SC vs. 6.06 for MC).
  • Negative Correlation: The researchers found a weak negative correlation (Pearson coefficients between -0.21 and -0.34) between modularity and performance. The extra "boilerplate" required for function definitions and calls may fragment the context or distract the model.
  • Neutral Perplexity: LLMs showed nearly identical perplexity for both styles, suggesting that pre-training data contains enough of both modular and non-modular code for the models to handle them with equal ease.

Conclusion

While modularity remains essential for human-centric software development, it is not a prerequisite for high-quality LLM code generation. In fact, non-modular "singular" code often serves as a more effective prompt or fine-tuning target. Future research should pivot from structural modularity toward other code properties—such as identifier naming and algorithmic patterns—to identify the true drivers of LLM performance in code generation tasks.

Summary updated

BibTeX
@inproceedings{kang2024revisiting,
  author = {Deokyeong Kang and Ki Jung Seo and Taeuk Kim},
  title = {{Revisiting the Impact of Pursuing Modularity for Code Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.676},
}

Loading related content...

Figure