Fast: Similarity-Based Knowledge Transfer for Efficient Policy Learning

Alessandro Capurso, Elia Piccoli, Davide Bacciu

FAST: Similarity-based Knowledge Transfer for Efficient Policy Learning

TL;DR

FAST (Framework for Adaptive Similarity-based Transfer) accelerates Reinforcement Learning by using a multimodal similarity metric—combining visual frames and textual descriptions—to select the best pre-trained "source" policies for a new task. By intelligently reusing existing knowledge, FAST can achieve performance comparable to training from scratch while using up to 80% fewer training steps.

What Problem Does This Paper Address?

Reinforcement Learning (RL) is notoriously sample-inefficient, often requiring millions of interactions with an environment to learn a simple policy. While Transfer Learning (TL) aims to solve this by reusing knowledge from previously solved tasks, it faces two major hurdles:

  1. Negative Transfer: If an agent tries to learn from a source task that is too different from the target task, its performance can actually decrease.
  2. Source Selection: There is no universally accepted way to quantify how "similar" two RL tasks are, making it difficult to choose which prior knowledge to leverage.

This is particularly problematic in evolving domains like game development, where environments change slightly between iterations, and re-training agents from scratch every time is computationally prohibitive.

Methods

The authors introduce FAST, which integrates a novel similarity metric directly into the RL training loop (specifically using Soft Actor-Critic (SAC) as the base algorithm).

  • Multimodal Task Embedding: To compare tasks, FAST creates a latent representation of the environment.
  • Visual: An autoencoder processes four consecutive environment frames into a compact vector.
  • Textual: Natural language descriptions of the task goals (e.g., "maintain the car on the roadway") are embedded using BERT.
  • Similarity Scoring: These visual and textual embeddings are concatenated and compared using cosine similarity.
  • The FAST Pipeline: Every $K$ timesteps during training, the agent calculates the similarity between its current task and a repository of pre-trained source policies. If the similarity score exceeds a threshold ($\Theta$), the agent adopts the most similar source policy to guide its actions for a set period. If no task is similar enough, the agent continues learning on its own to avoid negative transfer.

Datasets

The framework was evaluated using Highway-Env, a common benchmark for autonomous driving and tactical decision-making.

  • Source Tasks: Highway, Indiana (oval track), Intersection, Lane Centering, Merge, and Roundabout.
  • Target Tasks: Indiana, Racetrack (complex curves), and a newly introduced Custom Racetrack designed to be significantly longer and more difficult than standard tracks.
  • Dataset Generation: The authors generated a custom dataset of 6,000 examples (frames and descriptions) from these environments to train their autoencoder.

Key Insights and Ideas

  • Text Disambiguates Vision: One of the "aha" moments is that visual similarity alone isn't enough. Two environments might look identical (e.g., a straight road) but have different goals (e.g., "speed up" vs. "stay in lane"). Adding BERT-based textual descriptions helps the agent distinguish between these intents.
  • Efficiency through Sparsity: You don't need to check for similarity at every single step. By using a hyperparameter $K$ to check similarity periodically, the authors reduced computational overhead while still providing enough guidance to the agent.
  • The "One-Fifth" Rule: The most striking insight is the efficiency gain. In complex scenarios, the transfer-enabled agent reached the performance of a "from-scratch" baseline in just 2 million steps, whereas the baseline required 10 million steps to reach the same level.

Key Results

  • Performance Boost: On the "Indiana" track, the FAST variant using both frames and text (FT) achieved an average reward nearly double that of the baseline SAC agent.
  • Speedup: On the "Racetrack" environment, FAST achieved competitive final performance while requiring 5x fewer training steps than the baseline.
  • Robustness: The system successfully identified relevant source policies. For example, when training on a racetrack, it correctly prioritized knowledge from "Highway" and "Roundabout" tasks, which share similar driving dynamics.

Limitations and Future Work

  • Description Quality: The authors noted that the impact of textual descriptions was inconsistent across all tasks. They suggest that more precise, detailed descriptions might be necessary to fully unlock the potential of the textual embedding.
  • Similarity Metrics: While cosine similarity was effective, the authors plan to explore more sophisticated ways to compare latent representations.
  • Sequential Learning: The current work focuses on one-to-one transfer. Future work aims to integrate FAST into a Continual Learning setting, where an agent builds an ever-growing library of skills and autonomously decides when to reuse them across a long sequence of different tasks.

Summary updated

Abstract

Transfer Learning (TL) offers the potential to accelerate learning by transferring knowledge across tasks. However, it faces critical challenges such as negative transfer, domain adaptation and inefficiency in selecting solid source policies. These issues often represent critical problems in evolving domains, i.e. game development, where scenarios transform and agents must adapt. The continuous release of new agents is costly and inefficient. In this work we challenge the key issues in TL to improve knowledge transfer, agents performance across tasks and reduce computational costs. The proposed methodology, called FAST - Framework for Adaptive Similarity-based Transfer, leverages visual frames and textual descriptions to create a latent representation of tasks dynamics, that is exploited to estimate similarity between environments. The similarity scores guides our method in choosing candidate policies from which transfer abilities to simplify learning of novel tasks. Experimental results, over multiple racing tracks, demonstrate that FAST achieves competitive final performance compared to learning-from-scratch methods while requiring significantly less training steps. These findings highlight the potential of embedding-driven task similarity estimations.
BibTeX
@inproceedings{capurso2025fast,
  author = {Alessandro Capurso and Elia Piccoli and Davide Bacciu},
  title = {{Fast: Similarity-Based Knowledge Transfer for Efficient Policy Learning}},
  booktitle = {CoG},
  year = {2025},
  doi = {10.1109/cog64752.2025.11114355},
}

Loading related content...

Figure