Policy Transfer Across Heterogeneous State Action Spaces

123 concepts98 papers1 proposal

Standard reinforcement learning (RL) and imitation learning algorithms operate under a strict assumption of homogeneity: the agent's physical body, sensors, and actuators must remain identical across tasks. If you train a policy for a quadruped robot, that policy is mathematically useless for a bipedal robot. Their state spaces (e.g., joint angles, sensor configurations) and action spaces (e.g., motor torques, available moves) possess completely different dimensions and lack a common coordinate system.

As noted by Jin Zhang et al. and Hao Li et al., this structural mismatch forces agents to learn from scratch whenever hardware configurations or sensor modalities change. To overcome this limitation, a prominent line of research focuses on state-action space alignment. The goal is to build a functional bridge between heterogeneous domains so that knowledge gained by a "teacher" agent can be seamlessly transferred to a morphologically distinct "student" agent.

Learning Domain-Invariant Latent Spaces

How do we compare the incomparable? The core intuition behind many alignment methods is to project the unique state-action spaces of different robots into a shared, lower-dimensional latent space where their underlying behaviors mean the same thing.

Abhishek Gupta et al. demonstrated this by utilizing a set of simple "proxy tasks" that both robots already know how to perform. By observing how different morphologies execute these common tasks, the system learns a shared feature space that aligns their distinct coordinate systems.

Instead of relying on pre-existing proxy tasks, can we align these spaces completely unsupervised? Haitham Bou-Ammar introduced a framework that uses unsupervised manifold alignment to discover a shared latent space between completely different dimensionalities. This mathematical bridge allows an agent to "jumpstart" its learning in a high-dimensional target domain (like a quadrotor) using policies optimized in a low-dimensional source domain (like a cart-pole).

Source Space (e.g., Quadruped)  ---> [ Encoder A ] ---\
                                                       ---> [ Shared Latent Space ]
Target Space (e.g., Bipedal)    ---> [ Encoder B ] ---/

To ensure that this learned latent space preserves the core task dynamics rather than just geometric similarities, Michael Wan et al. proposed optimizing the embedding space using mutual information. By maximizing the mutual information between the teacher's internal neural representations and the student's latent states, the student effectively distills complex behaviors across structural mismatches. Similarly, Zhao-Heng Yin et al. leverage domain-invariant representations to enable cross-domain imitation learning without requiring tedious, human-labeled pairwise data.

Explicit State and Action Translation

Rather than finding a passive shared space, another strategy is to actively translate actions and states from one domain to another, acting like a language translator for robot control.

Yijie Guo et al. developed a context-conditioned action translator (MCAT) that maps actions from a source task directly to equivalent outcomes in a target task with different physics. This allows an agent to reuse successful strategies even when its actuators have different strengths, masses, or configurations.

When the mapping between controls isn't obvious, how do we discover it? Matthew E. Taylor et al. introduced Inter-Task Action Correlation (I-TAC), which automatically learns action mappings by training a classifier on state-transition data from both environments. This removes the need for human experts to manually specify which action on a new robot corresponds to an action on an old one.

Furthermore, Heng You et al. expanded this concept to multi-policy reuse. Their Cross-domain Adaptive Transfer (CAT) framework learns a latent state-action correspondence dynamically, allowing a target agent to adaptively weight and reuse policies from multiple distinct source agents (e.g., a four-legged robot learning from an eight-legged one) while avoiding the pitfall of "negative transfer" where bad advice hinders learning.

Aligning States and High-Level Abstractions

Sometimes, translating exact actions is impossible because the physical capabilities of the robots are fundamentally mismatched—a biped simply cannot execute a quadruped's gait. In these scenarios, the alignment must happen at the level of states or value functions rather than raw controls.

Fangchen Liu proposed State Alignment-based Imitation Learning (SAIL), which argues that a student does not need to copy a teacher's exact actions. Instead, by aligning the sequence of states (the trajectory of outcomes), a student with entirely different dynamics can figure out its own unique actions to achieve the same global results.

Teacher Trajectory:  [State S1] ---> [State S2] ---> [State S3]
                       |               |               |
                       v (Align)       v (Align)       v (Align)
Student Trajectory:  [State T1] ---> [State T2] ---> [State T3]
                       ^               ^               ^
                       |-- (Student computes its own actions to match)

At an even higher level of abstraction, we can transfer the underlying value functions or rules. Matthew E. Taylor et al. showed that mapping value functions across tasks with different state-action dimensions allows an agent to transfer structural knowledge from simple tasks to complex ones.

Matthew E. Taylor later demonstrated that agents can autonomously discover these inter-task mappings. This allows a policy or value function learned in a small, low-dimensional environment to jumpstart learning in a massive, high-dimensional target domain, proving that true generalization in reinforcement learning relies not on rigid physical replication, but on flexible structural alignment.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_state_action_space_alignment_0c007349,
  title = {state-action space alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-action-space-alignment/rdr_0c007349b47f4b209b342f9753446079},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_state_action_space_alignment_0c007349_v1,
  title = {state-action space alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-action-space-alignment/rdr_0c007349b47f4b209b342f9753446079/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake