When we watch an expert perform a task, we usually try to figure out what they are trying to achieve. In machine learning, this is the domain of Inverse Reinforcement Learning (IRL): observing behavior to recover an underlying reward function. However, standard IRL makes a dangerous assumption—that every action the expert takes is a move toward a goal.
In reality, expert behavior is shaped as much by what they are avoiding as by what they are seeking. A driver stays in their lane not because the lane lines are inherently rewarding, but because crossing them is forbidden. Dexter R. R. Scobee argues in Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning that we must shift our focus from what an agent wants to do to what an agent is forbidden from doing. This shift is the foundation of Inverse Constrained Reinforcement Learning (ICRL).
The Ambiguity of Avoidance
The fundamental problem in imitation is the "Ambiguity of Avoidance." If an expert avoids a certain state, is it because that state has a low reward, or because it is strictly prohibited?
As Andreas Schlaginhaufen and Maryam Kamgarpour demonstrate in Identifiability and Generalizability in Constrained Inverse Reinforcement Learning, if we conflate these two, our learned models fail to generalize. If an agent thinks a safety constraint is just a "low reward" area, it might decide to enter that dangerous area if the potential reward elsewhere becomes high enough. By decoupling the reward (the "why") from the constraint (the "rules"), ICRL allows agents to learn behaviors that remain safe even when the goal changes.
Inferring the Hidden Rules
How do we "see" a constraint that is never explicitly stated? The intuition, explored by Shehryar Malik and colleagues in Inverse Constrained Reinforcement Learning, lies in the discrepancy between nominal optimal behavior and expert behavior.
Imagine you know the reward function (e.g., "get to the finish line as fast as possible"). You can calculate the mathematically optimal path. If the expert takes a longer, seemingly "suboptimal" path, that gap suggests the existence of a hidden constraint. The expert is not being inefficient; they are satisfying a rule you haven't seen yet.
To scale this to complex environments, researchers have developed several specialized frameworks:
- Stochastic Environments: Mattijs Baert and Sam Leroux introduced MCE-ICRL, which uses maximum causal entropy to handle the noise and unpredictability of real-world movements (Maximum Causal Entropy Inverse Constrained Reinforcement Learning).
- Offline Learning: Guorui Quan and Guiliang Liu proposed ICSDICE, which learns constraints from static datasets by identifying "superior distributions"—behaviors that would get more reward but are never seen in the expert data, likely because they are unsafe (Learning Constraints from Offline Demonstrations via Superior Distribution Correction Estimation).
Constraints are Dynamics-Dependent
A common misconception is that a constraint is a physical property of the world, like the location of a wall. However, Gokul Swamy and Andrea Bajcsy reveal a deeper truth in Your Learned Constraint is Secretly a Backward Reachable Tube.
They argue that what we learn as a "constraint" is actually a Backward Reachable Tube (BRT): the set of states from which a collision is inevitable given the agent's specific physics. A heavy truck must start braking much earlier than a small car to avoid a wall. Therefore, the "forbidden zone" for the truck is much larger than for the car. This means learned constraints are not universal; they are conditioned on the agent's dynamics.
Robustness and Multi-Agent Systems
In the real world, we rarely have a perfect model of the environment. Sheng Xu and Guiliang Liu addressed this with AR-ICRL, a game-theoretic approach that treats model uncertainty as an adversary (Robust Inverse Constrained Reinforcement Learning under Model Misspecification). By assuming the environment might be slightly different than expected, the agent learns constraints that are robust to "misspecification."
Furthermore, these rules often involve multiple actors. Shicheng Liu and Minghui Zhu extended this to multi-agent settings with MA-BIRDS, allowing decentralized agents to reach a consensus on what the shared safety rules are while observing streaming data (Learning Multi-agent Behaviors from Distributed and Streaming Demonstrations).
Why It Works: The Power of Decoupling
The success of ICRL stems from its ability to treat safety as a first-class citizen. By explicitly modeling constraints separately from rewards:
- Interpretability: We can inspect the learned constraints to see if the AI has actually learned the "law" (e.g., "don't cross the red line").
- Transferability: If the goal changes (the reward function shifts), the safety constraints remain valid.
- Safety: It prevents the "reward hacking" common in standard RL, where an agent finds a high-reward path that is dangerously reckless.
As summarized in the Comprehensive Survey on Inverse Constrained Reinforcement Learning by Guiliang Liu et al., this field is moving toward a future where autonomous systems don't just mimic what we do, but understand the boundaries within which they must operate.