Symmetry and Equivariance in Reinforcement Learning

This direction is split into other directions: equivariant reinforcement learning. Its last version stays readable and citable here.

Version 1 of 1112 papers167 conceptsfrozen 2026-10-04

Standard Reinforcement Learning (RL) is notoriously data-hungry. An agent might spend millions of steps learning to pick up a block, only to fail completely if the block is rotated 90 degrees or if the robot is asked to use its left arm instead of its right. To a human, these tasks are identical; to a standard neural network, they are entirely different regions of a high-dimensional state space.

A growing body of research suggests that we can solve this "data hunger" by baking the geometric and structural symmetries of the physical world directly into the agent’s architecture. This approach, often framed through equivariance and MDP homomorphisms, allows agents to share experience across equivalent states, effectively multiplying the utility of every interaction.

The Core Duality: Invariant Values and Equivariant Actions

The intuition behind symmetry-aware RL rests on a fundamental distinction between how we evaluate a situation and how we act within it.

  1. Invariance for the Critic: If a robot is looking at a table, the "value" (expected future reward) of that state should not change if the entire table is rotated. The state is different, but the potential for success remains the same. Alexandru Cioba and colleagues demonstrate in Reinforcement Learning Using known Invariances that using invariant kernels allows an agent to update value estimates for all symmetric configurations simultaneously.
  2. Equivariance for the Actor: While the value is invariant, the optimal action must change predictably. If the target object moves 30 degrees to the left, the robot’s arm must also move 30 degrees to the left. Dian Wang and Robert Platt show in SO(2)-Equivariant Reinforcement Learning that by using equivariant neural networks, we can mathematically guarantee that if the input image rotates, the output action vector rotates in correspondence.

By enforcing this duality—an invariant critic and an equivariant actor—researchers like Buqing Nie have achieved dramatic results in humanoid locomotion. Their SE-Policy Nie et al. ensures bilateral consistency, preventing the "brain" of a robot from receiving conflicting signals for functionally identical left- and right-side movements.

Formalizing Symmetry via MDP Homomorphisms

To move beyond simple rotations, researchers use the framework of MDP Homomorphisms. This theory, popularized in the deep learning era by Elise van der Pol in MDP Homomorphic Networks, treats symmetry as a mapping from a complex environment to a simplified, abstract representation.

When an environment is homomorphic, multiple state-action pairs map to the same transition dynamics and rewards. By building networks that respect these mappings, agents don't just "see" a rotated board; they recognize it as the same underlying state. Prakash Panangaden and Doina Precup extended this to continuous spaces in Policy Gradient Methods in the Presence of Symmetries and State Abstractions, proving that updates calculated in the abstract space are valid for all symmetric states in the original environment.

Scaling to Planning and Multi-Agent Systems

The benefits of symmetry are not limited to model-free RL. In model-based planning, Equivariant MuZero demonstrates that if the transition and reward models are equivariant, the entire Monte Carlo Tree Search (MCTS) process inherits those symmetries. This means the agent can plan "once" for a specific configuration and have that plan remain valid across all transformed versions of that state.

In multi-agent settings, symmetry acts as a powerful regularizer. Joshua McClellan and Furong Huang introduced E2GN2 McClellan et al., which uses graph neural networks to exploit spatial symmetries in tasks like StarCraft II, achieving up to 5x better sample efficiency. Furthermore, Hengyuan Hu and Adam Lerer showed in "Other-Play" for Zero-Shot Coordination that forcing agents to be robust to symmetries prevents them from developing "secret languages" or arbitrary conventions, making them more compatible with human partners.

Handling the Real World: Partial Equivariance

A major hurdle for this field is that the real world is rarely perfectly symmetric. Gravity always points down, and a robot might have an obstacle to its left but not its right. Strict equivariance in these "symmetry-breaking" environments can actually hurt performance by forcing the agent to ignore important constraints.

To address this, Junwoo Chang and Jongeun Choi proposed Partially Equivariant Reinforcement Learning (PERL) Chang et al.. Their framework uses a gating mechanism to selectively apply equivariant updates only when the local environment justifies it. This allows the agent to benefit from the efficiency of symmetry while remaining grounded in the specific, asymmetric realities of its physical workspace.

Why It Works: The Search Space Reduction

The mathematical "why" behind these successes is the reduction of the search space. If an environment has a symmetry group $G$, a symmetry-aware agent effectively reduces the volume of the state-action space it must explore by a factor of $|G|$.

As Dian Wang demonstrates in On-Robot Policy Learning with O(2)-Equivariant SAC, this reduction is so significant that it allows robots to learn complex manipulation tasks directly on physical hardware in under two hours—a feat that typically requires days of training or massive simulated datasets. By hard-coding the "common sense" of geometry, we allow RL agents to focus their learning capacity on the logic of the task rather than the basic physics of the world.

Papers in this version

Showing 60 of 112 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_symmetry_and_equivariance_in_reinforceme_f1560b84,
  title = {Symmetry and Equivariance in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symmetry-and-equivariance-in-reinforcement-learning/rdr_f1560b848c5043228e7d41759f94d6a9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_symmetry_and_equivariance_in_reinforceme_f1560b84_v1,
  title = {Symmetry and Equivariance in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symmetry-and-equivariance-in-reinforcement-learning/rdr_f1560b848c5043228e7d41759f94d6a9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}