State Only Imitation Learning

658 concepts334 papers2 proposals

Imagine trying to learn how to drive a car purely by watching a video from the passenger seat. You can see the car turn, accelerate, and brake, and you can see the changing environment through the windshield. However, you cannot see the driver's feet on the pedals, nor can you feel the precise torque they apply to the steering wheel.

This scenario captures the core challenge of Imitation Learning from Observation (IfO), or state-only imitation learning. Traditional imitation learning relies on behavioral cloning, which requires complete state-action pairs $(s, a)$ to directly supervise an agent's policy. But the vast majority of data available in the world—such as millions of hours of YouTube videos, human demonstrations, or historical logs—is completely "action-free." We see the states (pixels, coordinates), but the underlying actions (motor commands, forces) are entirely hidden.

To scale robotic and autonomous systems, agents must learn to bridge this "action gap." Over recent years, researchers have developed three primary intuitions to solve this problem: inferring missing actions, matching state distributions directly, and learning implicit value functions from passive progression.

Inferring the Hidden Actions

The most intuitive way to handle missing actions is to guess them. If an agent can learn how the world's physics operate, it can look at a transition from state $s_t$ to state $s_{t+1}$ and infer the action $a_t$ that must have caused it.

This approach typically relies on training an inverse dynamics model $f_\phi(s_t, s_{t+1}) = \hat{a}_t$ using a small amount of the agent's own interactive environment data (where both states and actions are known). Once trained, this model acts as a translator, labeling passive expert videos with "pseudo-actions."

Passive Observation:  [State s_t] -------------------------> [State s_{t+1}]
                                     \                 /
                                  (Inverse Dynamics Model)
                                             v
Infered Supervision:                  [Pseudo-Action a_t]

For instance, Ramos introduces MAAD, which uses an inverse dynamics model to infer plausible expert actions, providing a strong regularization signal that stabilizes training and dramatically improves sample efficiency compared to pure reinforcement learning. Similarly, Liu et al. propose Decoupled Policy Optimization (DePO), which explicitly splits the policy into a high-level state planner ("where to go next") and a low-level inverse dynamics controller ("how to get there"). This separation allows the high-level intent to be transferred across entirely different robot embodiments, even if their physical motor controls differ.

Matching State Occupancy Distributions

What if inferring actions is too difficult or inaccurate? In highly complex or stochastic environments, a tiny error in an inferred action can compound rapidly, leading the agent astray.

An alternative intuition is to ignore actions entirely during the imitation phase. Instead of trying to copy what the expert did, the agent tries to be where the expert was. Mathematically, this means minimizing the divergence between the state occupancy distribution of the learner $\rho_\pi(s)$ and that of the expert $\rho_E(s)$. If the agent visits the exact same sequences of states as the expert, it will inherently achieve the same outcomes.

Expert Trajectory:  (State A) -> (State B) -> (State C) -> (Goal State)
                                   ^
Agent Policy:       (State A) -> (State B) -> (State D) -> [Divergence Penalty!]

This distribution-matching paradigm bypasses the action gap by turning imitation into an adversarial or variational optimization problem:

  • Adversarial Matching: Sun et al. developed FAIL (Forward Adversarial Imitation Learning), which matches state distributions step-by-step. By focusing purely on state transitions, FAIL provably eliminates the compounding error problem without requiring expert action labels.
  • Stationary Distribution Correction: Rather than running unstable adversarial games, Kim introduces LobsDICE, which reformulates offline learning from observation as a single convex optimization problem over stationary state-transition distributions, leveraging imperfect background data to bridge the action gap.
  • Optimal Transport: Schwing et al. propose PW-DICE, which minimizes the primal Wasserstein distance between the learner's and expert's state occupancies. This allows the system to use contrastively learned distance metrics, making the imitation process robust to visual shifts.

Value-Implicit Representations and Progress Meters

A third paradigm treats expert observations not as trajectories to copy, but as a topological map of success. When a human performs a task in a video, time flows monotonically toward a goal. Every frame represents progress.

If an agent can learn a visual representation where the "distance" between two image embeddings corresponds to the time or cost required to travel between them, it implicitly learns a reward function. The closer the agent's current state embedding is to the expert's final goal embedding, the higher the reward.

Expert Video Frames:  [Start] ------> [Midpoint] ------> [Goal]
Embedding Space:      Low Value ----> Medium Value ----> High Value

A prominent example of this is Value-Implicit Pre-training (VIP), introduced by Ma et al.. By framing passive video consumption as an action-free goal-conditioned reinforcement learning problem, VIP generates a smooth, continuous embedding space. The spatial distance to a goal image in this embedding space acts as a dense, universal reward signal that can guide downstream robotic tasks zero-shot.

Similarly, Bruce et al. introduce Explore Like Experts (ELE), which trains a "progress meter" on unlabeled gameplay videos. This internal sense of progression provides a crucial intrinsic reward that empowers agents to navigate sparse-reward environments where standard exploration completely fails.

Why It Matters: Scaling Beyond Teleoperation

The shift from state-action imitation to state-only imitation is fundamentally about data scale. Collecting state-action data typically requires tedious human teleoperation or specialized kinesthetic teaching. In contrast, state-only data is abundant.

By leveraging masked autoregressive modeling over sensorimotor sequences, Radosavovic demonstrates that a humanoid robot can learn real-world locomotion by training on action-free human videos alongside robot data, framing control as a next-token prediction problem. As these methods mature, the barrier between passive visual observation and active physical control continues to dissolve, allowing autonomous agents to learn almost anything simply by watching the world around them.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 370 papers
  2. Version 1 · 2026-10-04 · 411 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_state_only_imitation_learning_ca7f9d04,
  title = {state-only imitation learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-only-imitation-learning/rdr_ca7f9d041a60402db4a21999e8f0b2b4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_state_only_imitation_learning_ca7f9d04_v2,
  title = {state-only imitation learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-only-imitation-learning/rdr_ca7f9d041a60402db4a21999e8f0b2b4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake