TL;DR
Standard reinforcement learning (RL) aligns language models based on whether their final answer is correct, often encouraging shortcut behaviors like verbosity and unfaithful reasoning. Causally-Enhanced Policy Optimization (CE-PO) introduces an internal, differentiable reward that explicitly scores causal coherence along the generation pipeline—prompt ($Z$) $\to$ rationale ($X$) $\to$ answer ($Y$). By projecting out non-semantic shortcuts and blending coherence with accuracy via a power-mean combiner, CE-PO improves multi-step reasoning accuracy by an average of 5.49% without succumbing to length-hacking.
What Problem Does This Paper Address?
When large language models (LLMs) are trained with reinforcement learning (e.g., PPO or GRPO using outcome rewards), they are rewarded for what they generate rather than how they reached the conclusion. Consequently, models frequently learn to game proxy rewards—padding reasoning traces with unnecessary tokens, reciting memorized templates, or guessing correctly despite flawed reasoning steps.
Existing remedies typically require expensive human rationales, discriminative concept bottlenecks, or post-hoc decoding checks. What is missing is an annotation-free, model-internal training objective that penalizes spurious reasoning shortcuts and enforces that the prompt genuinely informs the rationale, and the rationale genuinely determines the final answer.
Methods
CE-PO introduces a drop-in reward-shaping framework compatible with standard policy optimization algorithms (PPO and GRPO):
- Base Jacobian Sensitivities: The authors quantify directed influence across $(Z \to X)$, $(X \to Y)$, and direct $(Z \to Y)$ using input-gradient Jacobians ($J_{AB} = \frac{\partial s_B}{\partial E(A)}$), where $s_B$ is the downstream log-likelihood and $E(A)$ is the upstream token embedding. The direct effect of $Z \to Y$ is isolated by projecting out the mediated path traversing $X$.
- Counterfactual Hardening: Raw gradients are easily inflated by surface-level features like sequence length and token frequency. CE-PO builds source-side counterfactuals by randomly permuting tokens (destroying semantics while strictly preserving token statistics). It computes counterfactual Jacobians to identify nuisance subspaces and projects them out of the base Jacobians via orthogonal projection.
- Spectral Normalization: The deconfounded Jacobian matrix is scalarized to $[0, 1]$ using its top singular value ratio (spectral energy share).
- Minkowski ($L_p$) Combiner: The continuous accuracy metric $r_{\text{base}}$ (e.g., BERTScore or a DeBERTa reward model) and the three causal scores $S(Z, X), S(X, Y), S(Z, Y)$ are fused using a weighted power mean: $$R_p = \left(w_0 r_{\text{base}}^p + w_1 S(Z, X)^p + w_2 S(X, Y)^p + w_3 S(Z, Y)^p\right)^{1/p}$$ Adjusting $p$ smoothly shifts optimization focus between accuracy ($p \to +\infty$) and coherence ($p \to -\infty$).
To maintain computational feasibility at scale, gradients are obtained via reverse-mode vector-Jacobian products (vJP) without materializing full Jacobian or Hessian matrices.
[ Prompt Z ]
/ \
v v
[ Rationale X ] ---> [ Answer Y ]
| |
+--- Jacobians ----+
|
[ Counterfactual Projection ] (Filters length/nuisance cues)
|
[ Minkowski Fusion L_p ] <--- Base Accuracy (BERTScore / RM)
|
Scalar Reward R_p ===> Plug into PPO / GRPO
Datasets
The authors utilized existing benchmarks formatted into question-answer pairs:
- Training & Validation: BBEHCausal (BIG-Bench Extra Hard causal split), CaseHOLD (legal holdings), MATHHARD (MATH level-5 competition problems), and IfQA (counterfactual open-domain QA).
- In-Domain Reasoning Evaluation: BBEHMATH (multistep arithmetic), CLadder (formal causal inference queries), LegalBench (reasoning in case law), and LogiQA (logical deductive exams).
- Out-of-Distribution (OOD) Generalization: Evaluated on 200 random samples each from TruthfulQA (factuality/misconceptions), CodeMMLU (programming knowledge), and SuperGPQA (expert STEM queries).
No new datasets were introduced.
Key Insights and Ideas
- The Causal Loop Must Stay Closed: Ablations demonstrate that dropping any link—$S(Z, X)$, $S(X, Y)$, or $S(Z, Y)$—triggers catastrophic reward degradation. Without the complete triangle constraint, the policy quickly exploits length shortcuts, inflating output token length to the cap.
- Counterfactuals Prevent Length-Hacking: Unhardened Jacobian rewards encourage the model to output vacuous, elongated tokens to artificially inflate gradient norms. Orthogonal projection against permuted counterfactuals prevents this without requiring arbitrary length penalty heuristics.
- Continuous Rewards Prevent Policy Oscillations: Replacing continuous rewards (BERTScore / DeBERTa) with discrete 0/1 accuracy judges induced unstable policy oscillations when combined with smooth causal scores.
- Internal Sensitivity Mirrors Human Truthfulness: A Welch’s $t$-test on generated outputs confirmed that correctly reasoned answers exhibited statistically significantly higher causal scores ($p < 0.01$) across all links compared to incorrect answers.
Key Results
- Performance Uplift: Tested across five open-source backbones (Qwen-3-1.7B-Thinking, Qwen-3-4B-Thinking, Phi-3.5-mini-Instruct, Llama-3.2-3B-Instruct, Llama-3-8B-Instruct), CE-PO variants outperformed non-causal RL baselines by 2.3 to 9.58 percentage points (average +5.49%).
- Model Efficiency & Footprint: On Qwen-3-14B, computing the hardened Jacobian reward added only 1.078× GPU memory overhead over standard BERTScore reward evaluation (peaking at 30.18 GB vs. 28.00 GB), with a wall-clock overhead of 5.82×.
- OOD Robustness: In zero-shot transfer, CE-PO delivered solid gains in structure-heavy domains like CodeMMLU and SuperGPQA while maintaining non-degraded factuality on TruthfulQA.
Limitations and Future Work
- Format Dependence: The approach relies on models cleanly partitioning rollouts into distinct rationale ($X$) and answer ($Y$) segments; base models without instruct/thinking fine-tuning struggle to follow this schema.
- Model Scale: Due to academic compute budgets, experiments were limited to models $\le 8\text{B}$ parameters (with throughput profiling up to 14B), leaving 70B+ model dynamics untested.
- Static Hyperparameters: The balance parameter $p$ and channel weights were fixed manually; the authors note that learning or dynamically scheduling the accuracy–causality trade-off during training is an open problem.
Summary updated