Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning

Jared Joselowitz, Ritam Majumdar, Arjun Jagota, Matthieu Bou, Nyal Patel, Satyapriya Krishna, Sonali Parbhoo

TL;DR

This paper introduces a novel framework that applies Maximum Margin Inverse Reinforcement Learning (IRL) to reverse-engineer the hidden reward functions of Large Language Models (LLMs) aligned via Reinforcement Learning from Human Feedback (RLHF). By treating an aligned model's outputs as "expert demonstrations," the authors successfully extract reward models that predict human preferences with up to 85% accuracy. This approach provides a powerful new diagnostic tool for auditing LLM safety and understanding the implicit goals driving model behavior.

What Problem Does This Paper Address?

While RLHF has become the industry standard for aligning LLMs with human values (e.g., making them less toxic or more helpful), the underlying reward functions the models learn remain a black box. This opacity raises significant safety concerns. For instance, a model might engage in "reward hacking"—exploiting spurious correlations to maximize its reward score without actually aligning with human intent.

Prior interpretability research has focused heavily on dissecting model architectures and internal representations, but these methods fail to expose the explicit incentives shaping LLM behavior. This paper addresses this gap by attempting to reverse-engineer the latent reward structures of aligned LLMs, making the training goals transparent and auditable.

Methods

The authors frame the problem through the lens of Inverse Reinforcement Learning (IRL), a technique traditionally used in robotics to infer an agent's hidden objectives by observing its actions.

Here is how they adapt it for LLMs:

  1. Expert Trajectories: They treat the text outputs generated by an RLHF-aligned LLM ($\pi_E$) as "expert demonstrations" and outputs from the unaligned base model ($\pi_{base}$) as suboptimal references.
  2. Maximum Margin IRL: They optimize a linear reward head ($R(o) = w^T h(o) + b$) parameterized by the hidden representations ($h(o)$) of the base LLM. The goal is to find a weight vector $w$ that maximizes the margin between preferred (non-toxic) and suboptimal (toxic) responses.
  3. Asymmetric Loss Function: To reflect the safety-critical nature of alignment, they implement an asymmetric max-margin loss. This loss penalizes the model much more severely if it incorrectly assigns a higher reward to a toxic output than to a non-toxic one.

Datasets

The framework was evaluated on the task of toxicity reduction using the following datasets:

  • Jigsaw Toxicity Dataset: A dataset of 2,000 sentences (balanced between toxic and non-toxic) used for training the ground-truth reward model, extracting the IRL rewards, and internal evaluation.
  • RealToxicityPrompts: A public benchmark used as an out-of-distribution dataset to evaluate the toxicity reduction performance of the models at test time.
  • Anthropic Helpful and Harmless (HH) Dataset: Used for the initial Supervised Fine-Tuning (SFT) of the base language models.

Key Insights and Ideas

  • Quality Over Quantity in Data: A small number of highly informative negative (toxic) samples provides a much stronger contrastive signal for learning rewards than a massive pool of redundant positive (non-toxic) data. In fact, adding too many redundant positive samples can actually degrade the precision of the recovered reward model.
Figure 2: Left (a, b, c): Increasing toxic examples in training improves precision, Kendall Tau, and recall, enhancing the model’s ability to rank non-toxic outputs. Right (d, e, f): Adding non-toxic data while keeping toxic samples fixed degrades classification and ranking quality, as precision and Kendall Tau decline, though recall remains high with slight variability.
  • Non-Identifiability as a Strength: In IRL, multiple distinct reward functions can often explain the same policy behavior (non-identifiability). Rather than viewing this as a flaw, the authors demonstrate that it captures a natural spectrum of human preferences—some reward models prioritize overall precision, while others focus heavily on penalizing extreme toxicity.
  • Scale Enhances Interpretability: Model scale plays a vital role. The larger Pythia-410M model captured reward function nuances significantly better and established a much clearer margin of separation between toxic and non-toxic outputs than the smaller Pythia-70M model.

Key Results

  • High Preference Accuracy: The IRL-extracted reward models achieved up to 85% accuracy and an 86.19% F1-score in predicting human preferences for the 410M parameter model, slightly outperforming the original ground-truth reward model at convergence.
  • Successful Downstream Alignment: The authors proved that the reverse-engineered reward models could be used to train entirely new LLMs via RLHF. The resulting models achieved comparable or even superior toxicity reduction on benchmarks. For the 70M model, the IRL-derived reward model reduced the Jigsaw toxicity ratio to 0.0405, outperforming the standard RLHF baseline (0.0419).
Scatter and violin plots showing reward separation
  • Robustness to Noise: The reward models extracted from well-aligned 410M models demonstrated remarkable resilience to Gaussian noise perturbations, maintaining stable classification performance even at higher noise standard deviations.

Limitations and Future Work

Despite its promise, the framework relies on two core assumptions: it requires direct access to token-level trajectories from the aligned model, and it assumes the base model's hidden representations are rich enough to serve as an effective feature map.

Additionally, the study was limited to relatively small models (70M and 410M parameters) focused on a single alignment axis (toxicity). The authors note that scaling this approach to multi-billion parameter models and evaluating it on more complex, multi-dimensional alignment objectives (like balancing helpfulness with harmlessness) are crucial next steps. Finally, they suggest exploring context-based loss functions to make the linear reward models less sensitive to hard syntactic perturbations.

Summary updated

Abstract

Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems.
BibTeX
@inproceedings{joselowitz2025insights,
  author = {Jared Joselowitz and Ritam Majumdar and Arjun Jagota and Matthieu Bou and Nyal Patel and Satyapriya Krishna and Sonali Parbhoo},
  title = {{Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning}},
  booktitle = {COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=Bs5Jb285qv},
}
Figure