Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse RL

Jared Joselowitz, Arjun Jagota, Satyapriya Krishna, Sonali Parbhoo

Abstract

Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems.

Introduction

Reinforcement Learning from Human Feedback (RLHF) is the cornerstone of modern Large Language Model (LLM) alignment, ensuring models are both helpful and safe. However, the resulting reward structures are often treated as "black boxes." While we can observe the final behavior, the specific incentives that guide the model remain hidden. This lack of transparency leads to critical issues such as "reward hacking"—where models find unintended shortcuts to high rewards—and a general lack of auditability for developers and regulators.

This paper proposes a method to bridge this gap by reverse-engineering the hidden goals of LLMs. By treating a trained LLM as an "expert" and applying Inverse Reinforcement Learning (IRL), we demonstrate that it is possible to recover the implicit reward functions that guide model behavior.

Methodology

The core of our approach is the application of Maximum Margin Inverse Reinforcement Learning to the LLM context. We treat an RLHF-trained model, specifically one aligned for toxicity reduction, as an expert agent. We then collect "expert trajectories," which are the text outputs generated by the model in response to various prompts.

To recover the reward function, we employ an Asymmetric Max-Margin Loss. This loss function is designed to reflect the safety-critical nature of alignment by penalizing the model more heavily when it incorrectly prefers a toxic output over a non-toxic one. We utilize the hidden representations (embeddings) from the base LLM as the features ($\phi$) for the reward function, assuming that the model's internal states must represent the concepts it has been trained to optimize, such as toxicity.

Results

Our experiments show that IRL-derived reward models can achieve high fidelity, with classification accuracies between 84% and 88.5%, closely matching the performance of original ground-truth reward models.

Distribution of IRL-extracted rewards for toxic and non-toxic categories.

As illustrated in the distribution of extracted rewards, the recovered model successfully differentiates between toxic and non-toxic content. This separation is further validated by comparing the extracted rewards against the ground truth rewards used in the original RLHF process.

Scatter plot comparing IRL-extracted rewards to ground truth rewards.

Key Insights

  • Non-Identifiability as a Feature: In IRL, multiple reward functions can often explain the same behavior. We argue this "non-identifiability" is a strength, allowing for a spectrum of reward models that capture different nuances of human preference.
  • Scale and Interpretability: We found that larger models (e.g., Pythia 410M) produced more stable and accurate reward reconstructions than smaller models (70M). This suggests that as models grow, their internal representations of alignment-relevant features become more robust.
  • Data Efficiency: Our results indicate that a small number of highly informative "negative" (toxic) examples are more valuable for learning a reward function than a large volume of "positive" examples.
  • Reward Transferability: Remarkably, a reward model extracted from one LLM can be used to successfully fine-tune a different model, achieving alignment comparable to using original human-labeled data.

Discussion and Limitations

While IRL provides a powerful tool for auditing LLMs, it is not without limitations. The method's success depends on the base model having a sufficiently rich feature map; if a concept is not represented in the embeddings, it cannot be recovered. Furthermore, the iterative nature of Max-Margin IRL is computationally intensive, and the approach requires access to the expert model's outputs.

Future work will explore more complex alignment goals beyond toxicity, such as factuality and reasoning, and investigate the use of IRL for detecting "sleeper agents" or other hidden vulnerabilities in LLMs. By reconstructing the underlying intent of these models, we can move towards more transparent and reliably aligned AI systems.

Summary updated

Abstract

Large language models (LLMs) trained with Reinforcement Learning from Human Feedback (RLHF) have demonstrated remarkable capabilities, but their underlying reward functions and decision-making processes remain opaque. This paper introduces a novel approach to interpreting LLMs by applying inverse reinforcement learning (IRL) to recover their implicit reward functions. We conduct experiments on toxicity-aligned LLMs of varying sizes, extracting reward models that achieve up to 85\% accuracy in predicting human preferences. Our analysis reveals key insights into the non-identifiability of reward functions, the relationship between model size and interpretability, and potential pitfalls in the RLHF process. We demonstrate that IRL-derived reward models can be used to fine-tune new LLMs, resulting in comparable or improved performance on toxicity benchmarks. This work provides a new lens for understanding and improving LLM alignment, with implications for the responsible development and deployment of these powerful systems.
BibTeX
@inproceedings{joselowitz2025insights,
  author = {Jared Joselowitz and Arjun Jagota and Satyapriya Krishna and Sonali Parbhoo},
  title = {{Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse RL}},
  booktitle = {COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=Bs5Jb285qv},
}
Figure