Abstract
Formal specifications are essential for ensuring software correctness, yet manually writing them is tedious and error-prone. Large Language Models (LLMs) have shown promise in generating such specifications from natural language intents, but the giant model size and high computational demands raise a fundamental question: Do we really need large models for this task? In this paper, we show that a small, fine-tuned language model can achieve high-quality postcondition generation with much lower computational costs. We construct a specialized dataset of prompts, reasoning logs, and postconditions, then supervise the fine-tuning of a 7B-parameter code model. Our approach tackles real-world repository dependencies and preserves pre-state information, allowing for expressive and accurate specifications. We evaluate the model on a benchmark of real-world Java bugs (Defects4J) and compare against both proprietary giants (e.g., GPT-4o) and open-source large models. Empirical results demonstrate that our compact model matches or outperforms significantly larger counterparts in syntax correctness, semantic correctness, and bug-distinguishing capability. These findings highlight that targeted fine-tuning on a modest dataset can enable small models to achieve results formerly seen only in massive, resource-heavy LLMs, offering a practical and efficient path for the real-world adoption of automated specification generation.
Introduction
Formal specifications—precise, verifiable conditions that describe how a program should behave—are the "holy grail" of software reliability. They enable automated bug finding, testing, and formal verification. However, writing them manually is a tedious, expert-level task that most developers avoid. While Large Language Models (LLMs) have shown promise in translating natural language into these specifications, state-of-the-art performance usually requires massive, proprietary models like GPT-4o. This creates a barrier for local, private deployment due to high computational costs. Furthermore, formal specification is a reasoning-heavy task where small models typically struggle with logical depth and complex repository dependencies.
Methods
We developed Qwen2.5-Coder-7B-Spec, a fine-tuned version of the 7B Qwen2.5-Coder model, to bridge this gap. The methodology is structured into three distinct stages as illustrated in Figure 1.
Our approach relies on three main pillars:
- Supervised Fine-Tuning (SFT) with Reasoning Traces: Instead of just training the model to output the final postcondition, we trained it to "think out loud." The training data includes reasoning logs where the model analyzes the method's intent and identifies potential pitfalls before generating the code.
- Repository-Level Context: Unlike previous methods that look at a single function in isolation, our Prompt Builder extracts "skeletons" of related classes and method signatures from the entire repository. This allows the model to understand how different parts of the code interact.
- Expressive Specification Format: We utilize a structured YAML format that supports "ghost variables" to capture the pre-state of an object (e.g., the size of a list before a method call), which is essential for defining meaningful postconditions.
Datasets and Evaluation
We curated a specialized dataset of approximately 1,500 triples consisting of a prompt, a reasoning trace, and a postcondition. This data was generated using DeepSeek-R1 and rigorously validated through automated testing and manual review. For evaluation, we utilized Defects4J, a standard benchmark of real-world Java bugs, deriving 1,134 generation tasks from 17 different Java projects including Apache Commons Math and JFreeChart.
Key Results
The fine-tuned 7B model achieved results that were competitive with GPT-4o across several metrics:
- Semantic Correctness (Sem@1): The model reached 43.0%, nearly matching GPT-4o (44.0%) and significantly outperforming DeepSeek-Coder-33B (32.6%), a model over four times its size. As shown in Figure 4, our model consistently outperforms the base Qwen2.5-Coder-7B and standard SFT baselines across various projects.
- Syntax Correctness: It achieved a 94.7% Syn@5 rate, meaning the generated specifications almost always compiled correctly within the existing codebase.
- Bug Distinguishing: The model successfully identified the behavioral difference between buggy and fixed code versions at a rate of 11.0%, comparable to GPT-4o's 11.8%.
Discussion and Insights
Our findings suggest that explicit reasoning allows the model to catch its own mistakes, acting as a form of self-correction. Furthermore, we discovered that high-quality, reasoning-enriched "tiny data" (only 1,500 samples) is sufficient to boost a 7B model's performance to state-of-the-art levels. However, we observed a "sweet spot" for reasoning length; performance peaks at a certain length and declines if the chains of thought become overly verbose.
Conclusion
This study demonstrates that massive scale is not a prerequisite for complex formal specification tasks. By leveraging repository-level context and explicit reasoning traces, a compact 7B model can rival proprietary giants. This offers a practical and efficient path for the real-world adoption of automated specification generation on consumer-grade hardware.
Summary updated