NLP+Code: Code Intelligence in Language Models

Terry Yue Zhuo, Qian Liu, Zijian Wang, Wasi Uddin Ahmad, Binyuan Hui, Loubna Ben AllalEMNLP 2025 · OpenReview

Abstract

Language models (LMs) like GPT and Claude have demonstrated remarkable proficiency in natural language processing (NLP), with code understanding and generation emerging as one of their most impactful applications. Given its nature as a set of executable logic forms, code offers a unique medium for exploring model capabilities. However, a gap remains in our practical understanding of how programming knowledge can be effectively combined with natural language to automate software development. Furthermore, while empirical studies suggest that code-centric training enhances complex reasoning and agentic automation, the underlying mechanisms of these improvements remain under-explored. This paper provides a coherent overview of "Code Intelligence"—the superior capabilities emerging from code modeling—covering foundational training, downstream evaluation, and the role of code in advancing general-purpose AI.

Introduction

The intersection of natural language processing and programming languages has birthed a new frontier in artificial intelligence: Code Intelligence. While Large Language Models (LLMs) have become proficient at generating snippets of Python or Java, the field is moving toward a more unified framework where code is treated as more than just a tool for developers. Instead, code is recognized as a highly structured, executable logic form that can serve as the backbone for complex reasoning and autonomous agents.

Defining Code Intelligence

Code Intelligence refers to the unique capabilities—logic, execution, and automation—that emerge when language models are exposed to the rigorous syntax and structural requirements of programming languages. Unlike natural language, which can be ambiguous, code requires strict adherence to logic to be functional. This tutorial-based overview explores how this structural rigidity translates into better multi-step planning and problem-solving across non-code domains.

The Lifecycle of Code-Centric Models

Foundational Training and Data

The development of high-performing code models begins with massive, curated datasets. Key resources include:

  • The Stack: A comprehensive dataset of source code used for large-scale pre-training.
  • StarCoder & StarCoder2: Open-access models that serve as benchmarks for the community, demonstrating the power of transparent, high-quality training data.
  • Cosmopedia & FineWeb-Edu: Synthetic and web-scale datasets used to refine a model's broader knowledge base and instructional following.

Evaluation Frameworks

As models evolve, so must our methods for testing them. Traditional metrics like "Pass@k" (which measures the probability of generating a correct solution in $k$ attempts) are being supplemented by more robust frameworks:

  • BigCodeBench & BigCodeArena: These benchmarks test the limits of code generation by requiring models to handle complex, real-world programming scenarios and long-context reasoning.
  • Execution-Based Evaluation: Moving beyond simple text matching to verify that the generated code actually runs and produces the correct output.

Code as a Reasoning Engine

One of the most significant insights in recent research is that training on code improves performance on general tasks, such as mathematical reasoning and multi-step planning. This is often referred to as the "Agentic Shift." Because code allows models to interact with environments—via API calls or script execution—it provides a framework for models to act as autonomous agents. The strict requirements of a compiler or interpreter force the model to maintain a higher degree of internal consistency than natural language alone might require.

Challenges and Future Directions

Despite the rapid progress of models like Qwen-Coder and the StarCoder series, several challenges persist:

  1. Practical Integration: We have yet to fully realize the potential of combining programming knowledge with natural language for seamless, end-to-end software engineering automation.
  2. Empirical Significance: While we observe that code improves reasoning, the community still lacks a definitive empirical explanation of why this transfer occurs.
  3. Alignment for Code: Ensuring that models not only write functional code but also secure, efficient, and maintainable code remains a primary research goal.

Conclusion

Code Intelligence represents a pivotal shift in how we train and utilize language models. By treating code as a fundamental logic form rather than a niche data type, researchers are unlocking new levels of reasoning and agency in AI. The future of the field lies in bridging the gap between engineering "how-to" and scientific "why," ultimately leading to models that can act as true partners in software development and complex problem-solving.

Summary updated

Abstract

Language models (LMs) like GPT and Claude have shown impressive abilities in a range of natural language processing (NLP) tasks. Among these tasks, code understanding and generation have quickly become one of the most popular applications of LMs, given its nature of executable logic forms. However, there is a practical understanding of how programming knowledge can be combined with natural language to automate software development. Moreover, recent studies also empirically demonstrate that code can be a better form for complex reasoning and agentic task automation, but they do not indicate their significance. In this tutorial, we deem such superior capabilities brought by code modeling as Code Intelligence, and aim to provide a coherent overview of recent advances in this topic. We will start by first providing preliminaries of training foundation models on code and their common practices. We will then focus on downstream tasks in the domain of code and their evaluations. Then, we will cover how code can contribute to advancements in general tasks, and the opportunities of future research on Code Intelligence.
BibTeX
@inproceedings{zhuo2025nlp,
  author = {Terry Yue Zhuo and Qian Liu and Zijian Wang and Wasi Uddin Ahmad and Binyuan Hui and Loubna Ben Allal},
  title = {{NLP+Code: Code Intelligence in Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=81ktGssaen},
}
Figure