Causally-Enhanced Reinforcement Policy Optimization of Large Language Models

Xiangqi Wang, Yue Huang, Yujun Zhou, Xiaonan Luo, Kehan Guo, Xiangliang ZhangSubmission to ICLR 2026 · Desk rejected

TL;DR

Standard reinforcement learning (RL) aligns language models based on whether their final answer is correct, often encouraging shortcut behaviors like verbosity and unfaithful reasoning. Causally-Enhanced Policy Optimization (CE-PO) introduces an internal, differentiable reward that explicitly scores causal coherence along the generation pipeline—prompt ($Z$) $\to$ rationale ($X$) $\to$ answer ($Y$). By projecting out non-semantic shortcuts and blending coherence with accuracy via a power-mean combiner, CE-PO improves multi-step reasoning accuracy by an average of 5.49% without succumbing to length-hacking.

What Problem Does This Paper Address?

When large language models (LLMs) are trained with reinforcement learning (e.g., PPO or GRPO using outcome rewards), they are rewarded for what they generate rather than how they reached the conclusion. Consequently, models frequently learn to game proxy rewards—padding reasoning traces with unnecessary tokens, reciting memorized templates, or guessing correctly despite flawed reasoning steps.

Existing remedies typically require expensive human rationales, discriminative concept bottlenecks, or post-hoc decoding checks. What is missing is an annotation-free, model-internal training objective that penalizes spurious reasoning shortcuts and enforces that the prompt genuinely informs the rationale, and the rationale genuinely determines the final answer.

Methods

CE-PO introduces a drop-in reward-shaping framework compatible with standard policy optimization algorithms (PPO and GRPO):

  1. Base Jacobian Sensitivities: The authors quantify directed influence across $(Z \to X)$, $(X \to Y)$, and direct $(Z \to Y)$ using input-gradient Jacobians ($J_{AB} = \frac{\partial s_B}{\partial E(A)}$), where $s_B$ is the downstream log-likelihood and $E(A)$ is the upstream token embedding. The direct effect of $Z \to Y$ is isolated by projecting out the mediated path traversing $X$.
  2. Counterfactual Hardening: Raw gradients are easily inflated by surface-level features like sequence length and token frequency. CE-PO builds source-side counterfactuals by randomly permuting tokens (destroying semantics while strictly preserving token statistics). It computes counterfactual Jacobians to identify nuisance subspaces and projects them out of the base Jacobians via orthogonal projection.
  3. Spectral Normalization: The deconfounded Jacobian matrix is scalarized to $[0, 1]$ using its top singular value ratio (spectral energy share).
  4. Minkowski ($L_p$) Combiner: The continuous accuracy metric $r_{\text{base}}$ (e.g., BERTScore or a DeBERTa reward model) and the three causal scores $S(Z, X), S(X, Y), S(Z, Y)$ are fused using a weighted power mean: $$R_p = \left(w_0 r_{\text{base}}^p + w_1 S(Z, X)^p + w_2 S(X, Y)^p + w_3 S(Z, Y)^p\right)^{1/p}$$ Adjusting $p$ smoothly shifts optimization focus between accuracy ($p \to +\infty$) and coherence ($p \to -\infty$).

To maintain computational feasibility at scale, gradients are obtained via reverse-mode vector-Jacobian products (vJP) without materializing full Jacobian or Hessian matrices.

       [ Prompt Z ] 
        /        \
       v          v
 [ Rationale X ] ---> [ Answer Y ]
       |                  |
       +--- Jacobians ----+
                 |
    [ Counterfactual Projection ] (Filters length/nuisance cues)
                 |
     [ Minkowski Fusion L_p ] <--- Base Accuracy (BERTScore / RM)
                 |
        Scalar Reward R_p  ===> Plug into PPO / GRPO

Datasets

The authors utilized existing benchmarks formatted into question-answer pairs:

  • Training & Validation: BBEHCausal (BIG-Bench Extra Hard causal split), CaseHOLD (legal holdings), MATHHARD (MATH level-5 competition problems), and IfQA (counterfactual open-domain QA).
  • In-Domain Reasoning Evaluation: BBEHMATH (multistep arithmetic), CLadder (formal causal inference queries), LegalBench (reasoning in case law), and LogiQA (logical deductive exams).
  • Out-of-Distribution (OOD) Generalization: Evaluated on 200 random samples each from TruthfulQA (factuality/misconceptions), CodeMMLU (programming knowledge), and SuperGPQA (expert STEM queries).

No new datasets were introduced.

Key Insights and Ideas

  • The Causal Loop Must Stay Closed: Ablations demonstrate that dropping any link—$S(Z, X)$, $S(X, Y)$, or $S(Z, Y)$—triggers catastrophic reward degradation. Without the complete triangle constraint, the policy quickly exploits length shortcuts, inflating output token length to the cap.
  • Counterfactuals Prevent Length-Hacking: Unhardened Jacobian rewards encourage the model to output vacuous, elongated tokens to artificially inflate gradient norms. Orthogonal projection against permuted counterfactuals prevents this without requiring arbitrary length penalty heuristics.
  • Continuous Rewards Prevent Policy Oscillations: Replacing continuous rewards (BERTScore / DeBERTa) with discrete 0/1 accuracy judges induced unstable policy oscillations when combined with smooth causal scores.
  • Internal Sensitivity Mirrors Human Truthfulness: A Welch’s $t$-test on generated outputs confirmed that correctly reasoned answers exhibited statistically significantly higher causal scores ($p < 0.01$) across all links compared to incorrect answers.

Key Results

  • Performance Uplift: Tested across five open-source backbones (Qwen-3-1.7B-Thinking, Qwen-3-4B-Thinking, Phi-3.5-mini-Instruct, Llama-3.2-3B-Instruct, Llama-3-8B-Instruct), CE-PO variants outperformed non-causal RL baselines by 2.3 to 9.58 percentage points (average +5.49%).
  • Model Efficiency & Footprint: On Qwen-3-14B, computing the hardened Jacobian reward added only 1.078× GPU memory overhead over standard BERTScore reward evaluation (peaking at 30.18 GB vs. 28.00 GB), with a wall-clock overhead of 5.82×.
  • OOD Robustness: In zero-shot transfer, CE-PO delivered solid gains in structure-heavy domains like CodeMMLU and SuperGPQA while maintaining non-degraded factuality on TruthfulQA.

Limitations and Future Work

  • Format Dependence: The approach relies on models cleanly partitioning rollouts into distinct rationale ($X$) and answer ($Y$) segments; base models without instruct/thinking fine-tuning struggle to follow this schema.
  • Model Scale: Due to academic compute budgets, experiments were limited to models $\le 8\text{B}$ parameters (with throughput profiling up to 14B), leaving 70B+ model dynamics untested.
  • Static Hyperparameters: The balance parameter $p$ and channel weights were fixed manually; the authors note that learning or dynamically scheduling the accuracy–causality trade-off during training is an open problem.

Summary updated

Abstract

Large language models (LLMs) trained with reinforcement objectives often achieve superficially correct answers via shortcut strategies, pairing correct outputs with spurious or unfaithful reasoning and degrading under small causal perturbations. We introduce Causally-Enhanced Policy Optimization (CE-PO), a drop-in reward-shaping framework that augments policy optimization with a differentiable proxy for causal coherence along the generation pathway from prompt ($Z$) to rationale ($X$) to answer ($Y$). CE-PO estimates model-internal influence with Jacobian-based sensitivities, counterfactually hardens these signals to suppress nuisance cues, and fuses the resulting coherence score with task-accuracy feedback via a Minkowski (power-mean) combiner, exposing a single tunable between accuracy and coherence trade-off. The unified reward integrates with PPO/GRPO without architectural changes. Across reasoning benchmarks and causal stress tests, CE-PO reduces reward hacking and unfaithful chain-of-thought while improving robustness to correlation--causation flips and light counterfactual edits, all at near-parity accuracy. Experimental results across 4 datasets show that CE-PO improves accuracy over baselines by 5.49 % on average (up to 9.58 %), while improving robustness to correlation–causation flips and light counterfactual edits.
BibTeX
@misc{wang2025causally,
  author = {Xiangqi Wang and Yue Huang and Yujun Zhou and Xiaonan Luo and Kehan Guo and Xiangliang Zhang},
  title = {{Causally-Enhanced Reinforcement Policy Optimization of Large Language Models}},
  year = {2025},
  eprint = {2509.23095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.23095},
}
Figure