Alignment Flywheel-Guided Suppression Patching for Interactionless Inverse Reinforcement Learning

Elias Malomgré, Pieter SimoensarXiv 2026 · arXiv

TL;DR

This paper introduces flywheel-guided patching, a surgical method for correcting learned reward models in reinforcement learning without the need for full retraining. By identifying problematic regions—such as where a model overgeneralizes or violates safety norms—and compiling these findings into local "suppression patches" in the model's latent space, the authors enable efficient, targeted alignment that scales to complex environments.

What Problem Does This Paper Address?

In Reinforcement Learning (RL), we often rely on learned reward models (via Inverse RL) because hand-designing scalar rewards is notoriously difficult. However, these learned models are prone to reward hacking and overgeneralization: an agent might find a "loophole" state that the model mistakenly scores highly, leading to unwanted behavior.

Furthermore, requirements often change after a model is deployed. You might discover a new safety constraint or a social norm that wasn't present in the original demonstration data. Traditionally, fixing this meant retraining the entire reward model from scratch—a process that is computationally expensive and risks "forgetting" useful learned behaviors. The challenge is: how do we update a reward surface locally and efficiently?

Methods

The authors propose a framework called the Alignment Flywheel to manage reward corrections. The process follows a specific pipeline:

  1. Red Teaming: An adaptive sampler searches the state space for "flaws"—states that receive high rewards but shouldn't.
  2. Verification: A verifier (which could be a simple script, a simulator, or even a human) confirms if the state is indeed problematic.
  3. Patch Construction: For every verified flaw, a local "suppression patch" is created.
  4. Regression Testing: Before committing, the patch is checked against known "good" (expert) states to ensure the correction doesn't accidentally suppress desired behavior.
Fig. 1. Flywheel-guided patching. Demonstrations train a base IIRL reward artifact. The Alignment Flywheel uses external judgments to identify regions where reward should be reduced, either to address overgeneralization or to incorporate new require- ments. Verified corrections are compiled into local suppression patches in latent space. These patches modify the reward mapping without retraining the learned evaluator.

Technically, the method is instantiated for Interactionless Inverse Reinforcement Learning (IIRL) using the MoE-GUIDE architecture (a Mixture-of-Experts autoencoder). Instead of updating the model weights ($E_\theta$), they modify the reward mapping function ($g$). The patches are placed in the model's latent bottleneck space, which is more compact and efficient than the raw state space.

To keep online evaluation fast, they use Locality-Sensitive Hashing (LSH). During training, the agent only retrieves and calculates suppression for a small subset of patches near its current latent state, rather than checking every single patch in the database.

Datasets

The authors evaluated their method across several benchmarks:

  • MuJoCo Benchmark: Standard environments including Swimmer, Hopper, Walker2d, HalfCheetah, and Ant.
  • Controlled Quadruped: A custom planar quadruped environment designed to test specific operator-specified norms (like lane discipline and wall clearance) that were absent from the training demonstrations.
  • Sparse Ant: A challenging version of the Ant environment where new constraints explicitly conflict with the original demonstration data.

Key Insights and Ideas

  • Compilation vs. Shielding: Unlike "safety shields" that check rules at every runtime step, patching "compiles" expensive or slow judgments (like human review or complex simulations) into a fast reward-lookup table. This allows the agent to learn to avoid bad regions during training.
  • Latent Space Advantage: Patching in the latent space is significantly more efficient than in the raw observation space. In high-dimensional tasks like Ant, a latent-space "ball cover" requires far fewer centers to cover the same problematic regions.
  • Soft Edits: Patching is a "soft" correction. It reduces the reward in specific regions rather than providing a hard binary constraint, which can help maintain training stability.
  • Handling Conflict: The method successfully improves performance even when new constraints (e.g., "don't tilt too much") conflict with the original demonstrations, showing that you can refine an agent's behavior beyond what the expert showed.

Key Results

  • Performance Gains: Flywheel Patching achieved the highest mean return in 3 out of 5 MuJoCo environments. The most significant improvements were seen in Ant (+842 points) and HalfCheetah (+445 points) compared to the unpatched baseline.
  • Efficiency: The LSH retrieval mechanism showed massive scalability. For a set of 1 million patches, LSH provided a 1,263x speedup over a brute-force search, making it feasible for real-time RL training.
  • Throughput: In the controlled quadruped setting, compiled patches allowed for higher task throughput at the same collision rate compared to hard-coded runtime rules.
  • Constraint Satisfaction: In the Sparse Ant task, patching increased the return from 2.89 to 5.22 while simultaneously reducing constraint violations.

Limitations and Future Work

  • Architecture Dependence: The current implementation is tied to the MoE-GUIDE architecture and its specific latent space. It remains to be seen how well this transfers to other reward model architectures.
  • Simple Verifiers: The MuJoCo experiments used a relatively simple "expert-envelope" verifier (checking coordinate-wise bounds). The authors note that the true power of the method lies in using richer, more expensive verifiers (like human-in-the-loop), which they leave for future work.
  • Sample Size: The main benchmarks used five seeds; the authors acknowledge that more runs would help assess smaller differences between methods more accurately.
  • No Guarantees: As a "soft" reward edit, patching does not provide the formal safety guarantees that a hard runtime shield might offer.

Summary updated

Abstract

Learned reward models allow reinforcement-learning agents to use demonstrations, implicit feedback, safety constraints, social norms, or other sources that do not provide a complete hand-designed scalar reward. However, a learned reward model can assign high reward to unwanted behavior. A policy can exploit these errors during training. Later requirements can also identify additional regions where the deployed reward should change. We introduce \emph{flywheel-guided patching}, a mechanism for making such localized corrections without retraining the learned evaluator. An Alignment Flywheel identifies overgeneralized or constraint-violating regions and compiles verified findings into suppression patches. We instantiate the mechanism for Interactionless Inverse Reinforcement Learning using MoE-GUIDE, placing patches in the model's latent space and retrieving only a small local subset during online reward evaluation. In the main MuJoCo benchmark, we evaluate patching without expert-state reset warm starts. We also test whether operator-specified norms absent from the demonstrations can be compiled into the same reward interface and compare this with enforcing the corresponding rules online. The results show that localized corrective information can modify an existing learned reward without providing a complete replacement reward. Compilation can reuse judgments that are expensive or unavailable during control. Cheap deterministic rules may be simpler to check directly at each step. Code for MoE-GUIDE, Alignment Flywheel, and flywheel patching is at https://github.com/decide-ugent/Alignment-Flywheel.
BibTeX
@misc{malomgre2026alignment,
  author = {Elias Malomgré and Pieter Simoens},
  title = {{Alignment Flywheel-Guided Suppression Patching for Interactionless Inverse Reinforcement Learning}},
  year = {2026},
  eprint = {2611.01322},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01322},
}
Figure