Abstract
Transfer learning in reinforcement learning (RL) typically requires manual intervention to map actions between source and target tasks. This paper introduces Inter-Task Action Correlation (I-TAC), an autonomous method that discovers these mappings by analyzing state-transition signatures. By training a classifier to recognize the environmental effects of actions, I-TAC enables agents to leverage prior knowledge in new, more complex environments without human-defined action translations. Experimental results in the RoboCup Keepaway domain demonstrate that I-TAC accurately identifies functional similarities between actions, significantly reducing the data required for effective transfer.
Introduction
Reinforcement Learning (RL) has achieved remarkable success in complex domains, yet the "tabula rasa" approach—training from scratch—remains prohibitively slow for many real-world applications. Transfer learning offers a solution by allowing agents to reuse knowledge from a related source task. However, a persistent bottleneck in this process is the "action mapping problem." When a source task (e.g., 3-on-2 soccer) and a target task (e.g., 4-on-3 soccer) have different action spaces, a human expert must manually specify which actions are equivalent. This paper addresses the need for autonomous action mapping, proposing a method that allows agents to discover these relationships independently.
The I-TAC Algorithm
The Inter-Task Action Correlation (I-TAC) algorithm treats action mapping as a classification problem based on environmental dynamics. The core insight is that while action labels may change, the fundamental effect of an action on the state space—its "signature"—remains consistent.
The I-TAC process involves five distinct steps:
- Source Experience Gathering: As the agent learns the source task, it records transitions consisting of the initial state ($s$), the action taken ($a$), and the resulting state ($s'$).
- Classifier Training: A classifier is trained on the source data to predict the action $a$ given a state transition $(s, s')$. This model learns the unique "fingerprint" of each source action.
- Target Experience Gathering: The agent performs a brief period of random exploration in the target task to collect transitions $(s_{target}, a_{target}, s'_{target})$.
- Cross-Task Projection: Using a provided state mapping function ($γ$), target transitions are translated into the source state representation.
- Mapping via Voting: For each target action, the agent uses the source classifier to predict which source action most closely matches the observed transitions. The target action is then mapped to the source action that receives the most "votes."
Experimental Evaluation: RoboCup Keepaway
The researchers evaluated I-TAC using the RoboCup Keepaway benchmark. The source task was a 3 vs. 2 scenario, while the target task was a more complex 4 vs. 3 scenario. In the 4 vs. 3 task, the agent has an additional action: "Pass to 3rd closest teammate." This setup tests the algorithm's ability to handle expanding action spaces and varying player counts.
Results and Discussion
The I-TAC method proved highly effective at identifying functional similarities between actions across tasks. As shown in the classification table below, the algorithm achieved near-perfect mapping for "Hold" and "Pass 1" (closest teammate).
A key finding is the algorithm's ability to identify "functional similarity" rather than just literal matches. For example, "Pass 3" in the 4v3 task (passing to the furthest teammate) was correctly mapped to "Pass 2" in the 3v2 task, which also represents the furthest pass in that context. Interestingly, "Pass 2" in the 4v3 task showed an equal correlation with "Pass 1" and "Pass 2" from the source task (133 votes each), reflecting its intermediate nature in the expanded player field.
The efficiency of I-TAC is a major advantage. While training an RL agent from scratch in these domains can take dozens of hours, I-TAC requires only about 1,000 experience tuples—roughly 10 minutes of simulation—to establish a reliable mapping.
Conclusion and Future Work
I-TAC significantly advances the autonomy of transfer learning by automating action mapping. By defining actions through their environmental consequences, the method provides a robust way to bridge different task complexities. Future research will focus on removing the reliance on a pre-defined state mapping ($γ$) and exploring how failed action attempts can be used to further refine the classifier's understanding of action intent.
Summary updated