In reinforcement learning (RL), an agent’s success depends on how it perceives its environment. Traditionally, we group states together based on visual similarity or spatial proximity—if two states look the same or are close in coordinate space, we treat them as similar. However, this intuition often fails. Two points on opposite sides of a thin wall are spatially close but functionally worlds apart; conversely, two visually distinct rooms might serve the exact same purpose in a navigation task.
A growing body of research suggests that the most effective way to simplify complex environments is to group states based on transition dynamics and reachability. By focusing on what an agent can do from a state rather than what the state looks like, these methods build abstractions that are more robust, interpretable, and efficient.
The Failure of Spatial Invariance
Standard deep RL architectures, like Value Iteration Networks (VINs), often rely on convolutions to plan. Convolutions assume "spatial invariance"—the idea that the rules of the world are the same everywhere. Li Zhang, Xin Li, and colleagues argue in Universal Value Iteration Networks that this is a fundamental limitation. In complex environments, action outcomes change based on location.
To solve this, they propose replacing rigid kernels with a learned "Dynamics Set." This mechanism automatically organizes states into clusters based on their transition behaviors. By grouping states that share underlying transition rules, the agent can generalize what it learns about movement in one region to other functionally related regions, even if they are spatially distant.
Functional Equivalence and Pseudometrics
If two states lead to the same future rewards and the same next states under the same actions, they are "functionally equivalent." This concept is central to PLOFF, an algorithm developed by Robert Dadashi, Olivier Pietquin, and others.
In offline RL, agents often "hallucinate" high rewards for transitions they haven't actually seen. PLOFF addresses this by learning a pseudometric—a task-specific distance measure. Unlike Euclidean distance, this metric measures how "close" states are based on the environment's rewards and dynamics. This ensures that physical barriers or reward-relevant boundaries are reflected in the geometry of the agent's latent space, preventing it from treating functionally distinct states as identical just because they appear similar.
Reachability as a Formal Bridge
How do we move from continuous, messy sensor data to the discrete, symbolic logic required for high-level planning? The answer lies in reachability analysis.
The GARA framework, introduced by Mehdi Zadem and colleagues, uses formal verification to analyze how sets of states transition to one another. Similarly, the STAR framework partitions state spaces into symbolic regions where all states within a region share the same transition possibilities to subsequent regions.
This approach provides a theoretically grounded bridge between low-level control and high-level strategy. By defining subgoals through reachability consistency, these methods ensure that the simplified "map" of the task accurately reflects the underlying system logic, making learning significantly more data-efficient.
Interpreting Strategy through Policy Dynamics
Grouping states by dynamics doesn't just help the agent; it also helps the human observer. Traditional explanation methods often group states by the immediate action an agent takes. However, Ronny Luss, Amit Dhurandhar, and Miao Liu argue in their work on Strategic State eXplanation (SSX) that this is insufficient for understanding intent.
SSX clusters states based on policy dynamics—where the agent is likely to go next over a longer horizon. By grouping states that lead toward the same strategic outcome or milestone, SSX creates "meta-states" that align with human mental models of task progression. This makes the agent's high-level strategy apparent, transforming a black-box policy into a series of interpretable functional phases.
Summary of the Intuition
The shift from visual/spatial abstraction to dynamics-based abstraction can be summarized by three core principles:
- Functional over Visual: States are similar if they behave the same way, not if they look the same.
- Reachability as Structure: The "topology" of a task is defined by which states can reach which others, forming the basis for symbolic subgoals.
- Dynamics-Aware Geometry: Effective latent spaces must respect the physical constraints and reward structures of the environment, often requiring learned pseudometrics rather than standard distance formulas.