Abstract
Although Large Language Models (LLMs) have become capable reasoners, the problem of faithfulness persists: their reasoning can contain errors and omissions that are difficult to detect and that may obscure biases in model outputs. To address this issue, we introduce Semi-Structured Reasoning Models (SSRMs), which are trained to produce semi-structured representations of reasoning. SSRMs generate reasoning traces in a non-executable Pythonic syntax that names each reasoning step and marks its inputs and outputs. This structure allows SSRM traces to be automatically audited to identify reasoning flaws. We evaluate three types of audits: hand-crafted structured reasoning audits, written in a domain-specific language (DSL) implemented in Python; LLM-generated structured reasoning audits; and learned typicality audits, which apply probabilistic models over reasoning traces. We show that all of these methods can be used to effectively flag probable reasoning errors. Importantly, the auditability of SSRMs does not appear to compromise overall accuracy: in evaluation on twelve benchmarks and two model families, SSRMs demonstrate strong performance and generalizability relative to other models of comparable size.
Introduction: The Faithfulness Problem
Large Language Models (LLMs) are increasingly capable of complex reasoning via "Chain-of-Thought" (CoT) prompting. However, they suffer from a significant faithfulness problem: a model might provide the correct final answer while its internal reasoning steps are hallucinated, biased, or logically disconnected. In high-stakes domains like medicine, these "unfaithful" traces are dangerous because they obscure the model's actual decision-making process. Analyzing arbitrary, free-form text reasoning is difficult to scale, prompting the need for a format that is easy for machines to verify without compromising model intelligence.
Methodology: Semi-Structured Reasoning Models (SSRMs)
Researchers at Carnegie Mellon University have introduced Semi-Structured Reasoning Models (SSRMs), which generate reasoning traces in a Python-like syntax. The authors propose a two-stage training recipe to create these models:
- Supervised Fine-Tuning (SFT): The model is trained to produce reasoning traces in a "Pythonic" syntax that names each step (e.g.,
extract_patient_data) and explicitly marks inputs and outputs. These steps are non-executable structured text, allowing the model to handle complex concepts that standard code cannot easily represent. - Reinforcement Learning with Verifiable Rewards (RLVR): Using Group Relative Policy Optimization (GRPO), the model is rewarded for both correct answers and adherence to the required semi-structured format.
Auditing Reasoning Traces
Once trained, these structured traces can be automatically audited to identify logical errors. The study explores three primary auditing methods:
- Hand-crafted Audits: Python unit tests written by experts to check specific task logic, such as ensuring units were checked for every measurement.
- LLM-generated Audits: Utilizing a separate model (e.g., Claude 3.5) to automatically generate these unit tests.
- Typicality Audits: Using probabilistic models to analyze reasoning patterns. If a model’s sequence of steps is statistically unusual compared to successful examples, it is flagged as a likely error.
Key Results and Performance
Contrary to the common fear that strict formatting imposes a performance "tax," SSRMs actually achieved higher accuracy than unstructured CoT models of the same size. The Qwen-based SSRM achieved an overall average accuracy of 61.7% across all benchmarks, outperforming the identically trained unstructured baseline (60.8%).
On the challenging MedCalcV2 Rules task, the SSRM reached 43.2% accuracy, significantly beating the unstructured baseline's 38.4%. Furthermore, the audits were highly discriminative: on medical rules, traces that passed the audits had an accuracy of 46-48%, while those that failed dropped to 21-26%.
Discussion and Limitations
While SSRMs offer a path to more transparent AI, several limitations remain. Audits, like software unit tests, can only catch errors they are designed to detect; a model might pass all audits but still make a subtle semantic error within a single step. Additionally, the semi-structured format is more verbose than free-form text, leading to higher token overhead and potential increases in latency.
Future work will explore more efficient variable referencing to reduce token usage and investigate how these audits can be integrated directly into the reinforcement learning phase to teach models to avoid common logical pitfalls from the outset.
Summary updated