Inter-Task Action Correlation for Reinforcement Learning Tasks

Matthew E. Taylor, Peter Stone

Abstract

Transfer learning in reinforcement learning (RL) typically requires manual intervention to map actions between source and target tasks. This paper introduces Inter-Task Action Correlation (I-TAC), an autonomous method that discovers these mappings by analyzing state-transition signatures. By training a classifier to recognize the environmental effects of actions, I-TAC enables agents to leverage prior knowledge in new, more complex environments without human-defined action translations. Experimental results in the RoboCup Keepaway domain demonstrate that I-TAC accurately identifies functional similarities between actions, significantly reducing the data required for effective transfer.

Introduction

Reinforcement Learning (RL) has achieved remarkable success in complex domains, yet the "tabula rasa" approach—training from scratch—remains prohibitively slow for many real-world applications. Transfer learning offers a solution by allowing agents to reuse knowledge from a related source task. However, a persistent bottleneck in this process is the "action mapping problem." When a source task (e.g., 3-on-2 soccer) and a target task (e.g., 4-on-3 soccer) have different action spaces, a human expert must manually specify which actions are equivalent. This paper addresses the need for autonomous action mapping, proposing a method that allows agents to discover these relationships independently.

The I-TAC Algorithm

The Inter-Task Action Correlation (I-TAC) algorithm treats action mapping as a classification problem based on environmental dynamics. The core insight is that while action labels may change, the fundamental effect of an action on the state space—its "signature"—remains consistent.

The I-TAC process involves five distinct steps:

  1. Source Experience Gathering: As the agent learns the source task, it records transitions consisting of the initial state ($s$), the action taken ($a$), and the resulting state ($s'$).
  2. Classifier Training: A classifier is trained on the source data to predict the action $a$ given a state transition $(s, s')$. This model learns the unique "fingerprint" of each source action.
  3. Target Experience Gathering: The agent performs a brief period of random exploration in the target task to collect transitions $(s_{target}, a_{target}, s'_{target})$.
  4. Cross-Task Projection: Using a provided state mapping function ($γ$), target transitions are translated into the source state representation.
  5. Mapping via Voting: For each target action, the agent uses the source classifier to predict which source action most closely matches the observed transitions. The target action is then mapped to the source action that receives the most "votes."

Experimental Evaluation: RoboCup Keepaway

The researchers evaluated I-TAC using the RoboCup Keepaway benchmark. The source task was a 3 vs. 2 scenario, while the target task was a more complex 4 vs. 3 scenario. In the 4 vs. 3 task, the agent has an additional action: "Pass to 3rd closest teammate." This setup tests the algorithm's ability to handle expanding action spaces and varying player counts.

Results and Discussion

The I-TAC method proved highly effective at identifying functional similarities between actions across tasks. As shown in the classification table below, the algorithm achieved near-perfect mapping for "Hold" and "Pass 1" (closest teammate).

Classification results for mapping 4 vs. 3 target actions to 3 vs. 2 source actions.

A key finding is the algorithm's ability to identify "functional similarity" rather than just literal matches. For example, "Pass 3" in the 4v3 task (passing to the furthest teammate) was correctly mapped to "Pass 2" in the 3v2 task, which also represents the furthest pass in that context. Interestingly, "Pass 2" in the 4v3 task showed an equal correlation with "Pass 1" and "Pass 2" from the source task (133 votes each), reflecting its intermediate nature in the expanded player field.

The efficiency of I-TAC is a major advantage. While training an RL agent from scratch in these domains can take dozens of hours, I-TAC requires only about 1,000 experience tuples—roughly 10 minutes of simulation—to establish a reliable mapping.

Conclusion and Future Work

I-TAC significantly advances the autonomy of transfer learning by automating action mapping. By defining actions through their environmental consequences, the method provides a robust way to bridge different task complexities. Future research will focus on removing the reliance on a pre-defined state mapping ($γ$) and exploring how failed action attempts can be used to further refine the classifier's understanding of action intent.

Summary updated

Abstract

Introduction Reinforcement learning (RL) problems (Sutton & Barto 1998) are characterized by agents making decisions attempting to maximize total reward, which may be time delayed. RL problems contrast with classical planning problems in that agents do not know a priori how their actions will affect the world. RL differs from supervised learning because agents are never given training examples with the correct action labeled. An RL agent operating successfully in an environment could have its available actions disabled, degraded, or even augmented. Instead of learning from scratch with its new action set, we would like the agent to identify which of the actions in the new set, A′, are most similar to the actions in the original set, A, and then leverage its existing knowledge. In the general problem of transfer learning, a learner trains in a source task and then utilizes its learned knowledge to speed up learning or increase performance in a target task. In this paper, we consider pairs of tasks with different action sets. We present Inter-Task Action Correlation (I-TAC), a method for identifying actions in the target task which are most similar to actions in the source task. This mapping can then be used to automate knowledge transfer by identifying similarities between pairs of RL tasks. We evaluate our algorithm by fully implementing and testing it in the RoboCupsoccer Keepaway domain.
BibTeX
@inproceedings{taylor2006inter,
  author = {Matthew E. Taylor and Peter Stone},
  title = {{Inter-Task Action Correlation for Reinforcement Learning Tasks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure