Introduction
Imitation Learning (IL) has emerged as a primary method for training robotic systems when explicit reward functions are difficult to define. By observing an expert, a student agent learns to map states to actions. However, IL is fundamentally limited by the "compounding error" problem: small execution errors lead the agent into states outside the expert's training distribution, causing further errors and eventual task failure. This phenomenon, known as covariate shift, typically results in a sample complexity that grows quadratically ($T^2$) with the task horizon $T$.
This research addresses the fundamental question: why are some expert behaviors easier to imitate than others? By introducing a formal metric for expert stability, this work provides the first mathematical proof that the stability of the expert directly dictates the sample complexity of the imitation task.
Quantifying Stability: Incremental Gain Stability (IGS)
The authors introduce Incremental Gain Stability (IGS) as a framework to quantify the "easiness" of a task. IGS measures how a system responds to perturbations:
- Contracting Systems: These systems naturally pull back toward the original trajectory after a nudge. In these cases, the stability parameter $a$ equals 1.
- Degrading Systems: These systems eventually recover from perturbations but do so more slowly, represented by higher values of $a$.
By framing imitation through the lens of IGS, the authors show that stability acts as a buffer against the compounding errors that typically plague IL.
Algorithmic Approaches
To leverage these stability properties, the paper proposes two primary algorithms that enforce stability constraints during the learning process:
- IGS-BC (Behavior Cloning): A single-epoch learner that constrains the optimization to ensure the resulting policy is itself IGS-stable. This prevents the learner from adopting "brittle" policies that might perform well on the training data but fail under minor distribution shifts.
- CMILe (Constrained Mixing Iterative Learning): An iterative, DAgger-like algorithm that mixes the learned policy with the expert's policy over multiple rounds. CMILe incorporates two critical constraints: a trust-region constraint to prevent drastic policy updates and an IGS-stability constraint to maintain the dynamical integrity of the agent.
Theoretical Insights
The central contribution of this work is the mathematical link between the stability parameter ($a$) and the task horizon ($T$). The authors prove that for a contracting system ($a=1$), the dependence on $T$ in sample complexity bounds vanishes entirely. Specifically, the required number of trajectories $m$ scales as:
$$m \approx \epsilon^{-2a} \cdot T^{2a(1-1/a^2)}$$
When $a=1$, this simplifies to $m \approx \epsilon^{-2}$, meaning a near-infinite horizon task can be learned with a fixed number of samples. This result demonstrates that sublinear—or even horizon-independent—imitation is possible if the expert is sufficiently stable. Furthermore, the authors prove that without enforcing these stability constraints, it is mathematically impossible to achieve such efficient error bounds in certain linear systems.
Experimental Validation
The framework was evaluated across two distinct environments:
- Synthetic Nonlinear System: Using a 10-dimensional tunable system, the authors varied a stability parameter $p$. As predicted by the theory, imitation loss and goal error increased proportionally as the system became less stable.
- Unitree Laikago Quadruped: In a high-dimensional (18 DOF) PyBullet simulation, the authors tested a sideways walking task. The stability-constrained algorithms (CMILe+IGS) significantly outperformed vanilla Behavior Cloning and standard CMILe, particularly in low-data regimes. This suggests that stability constraints are most valuable when demonstrations are scarce.
Conclusion and Future Work
This research provides a rigorous theoretical foundation for the intuition that stable experts are easier to follow. By enforcing IGS constraints, practitioners can bypass the $T^2$ complexity bottleneck of standard imitation learning.
While the results are promising, the authors note a gap in the current theory: while CMILe performs better empirically, its theoretical bounds are currently looser than those for IGS-BC. Additionally, the current implementation assumes known dynamics to enforce the IGS constraint. Future research will focus on extending this framework to model-free settings and exploring its applications in safe reinforcement learning.
Summary updated