Abstract
Large Language Models (LLMs) frequently encounter difficulties when solving domain-specific calculation problems, such as those found in legal or medical contexts. These failures often stem from an inability to consistently follow complex, conditional rules embedded in professional documentation. We introduce Knowledge-Intensive Programs Generator (KIPG), a pipeline that translates domain documents into executable Python programs. By shifting the burden of logic from auto-regressive generation to a structured interpreter, KIPG significantly improves accuracy and consistency. Our experiments demonstrate that a 7B-parameter model equipped with KIPG can outperform models 30 times its size, achieving state-of-the-art results on specialized legal and medical benchmarks.
Introduction
While Large Language Models (LLMs) excel in general-purpose dialogue, their performance in professional calculation tasks—such as determining litigation fees or medical dosages—remains inconsistent. The primary challenge is not arithmetic itself, but rather logical consistency. Professional rules often involve intricate "if-then" conditions that LLMs may overlook or hallucinate during standard text generation.
Existing approaches like "Program of Thought" (PoT) generate code tailored to a specific user query. However, this method is often unstable because the model must reason about the domain rules and the query simultaneously. To address this, we propose decoupling domain logic from the query. By "codifying" domain knowledge into a reusable, rigid program structure before addressing specific questions, we can ensure the underlying rules are strictly followed.
Methodology
The KIPG pipeline consists of four distinct stages designed to move logic from the LLM's "imagination" to a Python interpreter:
- Offline Program Generation: A generator model ($\theta_G$) processes a domain document (e.g., a legal article) and produces a comprehensive Python program representing all internal rules and conditions.
- Variable Extraction: Upon receiving a user query, an extraction model ($\theta_E$) identifies relevant variables (e.g., "litigation amount," "years of employment").
- Execution: The system executes the pre-generated program using the extracted variables to obtain a precise, domain-compliant result.
- Conclusion Step: A final model ($\theta_C$) synthesizes the query, the source document, and the program output into a natural language response.
To ensure the reliability of the code generator, we employ Iterative Direct Preference Optimization (DPO). Unlike standard RLHF, which relies on subjective human feedback, KIPG uses execution correctness as a ground-truth signal. The model is trained to prefer programs that yield the correct answer for a set of training queries, effectively aligning the generator with the objective logic of the source text.
Datasets
We evaluated KIPG using two primary benchmarks:
- Legal Domain Dataset: A high-quality dataset comprising 1,950 instances derived from 591 Chinese law articles, covering categories such as Compensation, Tax, and Litigation Fees. This was developed with legal experts to ensure logical rigor.
- Medical Domain Dataset: A cross-domain test set involving medication instructions and clinical indicators to assess the system's ability to generalize to new professional fields.
Results and Discussion
KIPG demonstrates significant performance gains over traditional reasoning methods and larger models. On the legal dataset, a 7B-parameter model using KIPG achieved an accuracy of 69.86%, surpassing the 200B-parameter Qwen-Max (63.85%) and the previous best baseline, EEDP (61.07%). When scaled with a 72B model for the final conclusion step, accuracy reached 74.84%.
Key insights from our evaluation include:
- Decoupling Logic: Turning documents into programs prevents "query bias," where specific phrasing might lead an LLM to ignore crucial legal constraints.
- Zero-Shot Generalization: The generator trained on legal texts successfully produced effective programs for medical documents without additional training, suggesting it learned a meta-skill for translating natural language rules into code.
- Robustness: KIPG performed well on out-of-distribution categories like "Penalties" and "Traffic Violations," which were not present during training.
Limitations and Future Work
Despite its strengths, KIPG faces several challenges. In real-world Retrieval-Augmented Generation (RAG) scenarios where the correct document must be retrieved first, performance drops by approximately 10-12%. Additionally, the system is sensitive to extraction errors; if the model fails to identify the correct variables from a casual user query, the program will execute on incorrect data. Finally, accuracy tends to decrease as program complexity (line count) increases, highlighting a need for better handling of multi-article legal dependencies.
Conclusion
KIPG offers a robust framework for solving domain-specific calculation problems by formalizing natural language rules into executable code. By prioritizing logical consistency and leveraging execution-based optimization, KIPG enables smaller models to achieve professional-grade accuracy in complex fields like law and medicine.
Summary updated