Abstract
Reinforcement Learning (RL) is often hindered by extreme sample inefficiency and the computational cost of training agents from scratch for every new task. Transfer Learning (TL) offers a solution by reusing knowledge from prior tasks, yet it faces challenges like negative transfer and the difficulty of source policy selection. This paper introduces FAST (Framework for Adaptive Similarity-based Transfer), a methodology that leverages multimodal embeddings—combining visual frames and textual descriptions—to quantify task similarity. By selecting the most relevant source policies, FAST achieves performance parity with baseline methods while requiring up to 80% fewer training steps.
1. Introduction
In dynamic environments such as game development or autonomous driving, agents frequently encounter new scenarios that are variations of previously solved tasks. Standard RL approaches typically treat these as entirely new problems, leading to redundant computation. While Transfer Learning aims to bridge this gap, the lack of a robust metric for task similarity often leads to "negative transfer," where irrelevant knowledge degrades the agent's performance. FAST addresses this by creating a latent representation of task dynamics to guide the selection of source policies.
2. Methodology
2.1 Multimodal Task Embedding
FAST utilizes two distinct data streams to characterize an environment:
- Visual Representation: An autoencoder processes a stack of four consecutive environment frames. This captures the spatial and temporal dynamics of the task in a compact latent vector.
- Textual Representation: Natural language descriptions (e.g., "maintain the car on the roadway") are processed through a pre-trained BERT model. This provides high-level semantic context that visual data alone may lack.
2.2 Similarity-based Policy Selection
The visual and textual embeddings are concatenated to form a comprehensive task signature. The similarity between the current target task and a library of pre-trained source tasks is calculated using cosine similarity.
The framework integrates with the Soft Actor-Critic (SAC) algorithm. Every $K$ timesteps, the agent evaluates the similarity score. If the score exceeds a predefined threshold ($\Theta$), the agent adopts the policy of the most similar source task for a fixed duration. This periodic check ensures the agent benefits from external knowledge without the overhead of constant computation.
3. Experimental Setup
The framework was evaluated using the Highway-Env benchmark, a suite for autonomous driving and tactical decision-making.
- Source Tasks: Included standard scenarios like Highway, Indiana (oval), Intersection, Lane Centering, Merge, and Roundabout.
- Target Tasks: Focused on complex environments including Indiana, Racetrack, and a custom-designed "Long Racetrack" intended to test the limits of the agent's adaptation.
- Data: A dataset of 6,000 frame-description pairs was used to train the multimodal embedding space.
4. Results and Discussion
4.1 Efficiency and Speedup
The most significant finding was the reduction in training time. In the complex "Racetrack" environment, the FAST-enabled agent reached the performance ceiling of a from-scratch baseline in just 2 million steps, whereas the baseline required 10 million steps. This represents a 5x speedup (80% reduction in training interactions).
4.2 Performance Gains
On the "Indiana" track, the multimodal variant (using both frames and text) achieved an average reward nearly double that of the baseline SAC agent. The system demonstrated an intuitive ability to select source policies; for instance, when navigating a racetrack, it correctly prioritized knowledge from the "Highway" and "Roundabout" tasks, which share similar high-speed steering dynamics.
4.3 The Role of Textual Disambiguation
Experimental results highlighted that visual data alone can be misleading. Two environments may appear visually similar while having different objectives. The inclusion of BERT-based textual descriptions allowed the agent to distinguish between these intents, effectively mitigating the risk of negative transfer.
5. Conclusion
FAST demonstrates that embedding-driven task similarity is a powerful tool for efficient policy learning. By intelligently reusing existing knowledge, the framework significantly lowers the barrier for deploying RL agents in evolving domains. Future work will explore more sophisticated similarity metrics and the integration of FAST into continual learning settings, where agents must manage an ever-growing library of skills.
Summary updated