Introduction
This paper introduces a novel transfer learning approach for "long-lived" agents that solve a sequence of different tasks. Instead of transferring policies or value functions as is traditional, the authors propose learning a Reward Mapping Function. This function translates a new task's objective into an optimized "internal" reward signal, helping computationally limited agents perform better right from the start of a new task.
The Challenge of Bounded, Long-Lived Agents
In reinforcement learning (RL), we usually assume the reward function provided by the environment is the best signal to guide an agent. However, for "bounded" agents—those with limited memory or planning time—the objective reward might be too sparse or misleading to lead to good behavior quickly.
The challenge is even greater for long-lived agents that face a sequence of different tasks. While traditional transfer learning focuses on reusing policies (how to act) or value functions (how good a state is), these methods often struggle when the environment's dynamics or the goals change significantly. The authors identify a critical gap: how can an agent transfer the "wisdom" of what kind of internal guidance worked best in the past to help it tackle a brand-new objective?
Methodology: Reward Mapping Transfer
The authors build on the Optimal Rewards Problem (ORP), which distinguishes between the objective reward (what the designer wants) and the internal reward (the signal actually used to guide the agent's planning).
The core contribution is the Reward Mapping-Transfer ORP Agent. Its workflow operates as follows:
- The Mapping Function ($f$): The agent maintains a function that maps objective reward parameters ($\theta^o$) to initial internal reward parameters ($\theta^i$).
- Initialization: When a new task starts, the agent uses this mapping to "guess" a high-quality starting internal reward.
- Online Adaptation (PGRD): While performing the task, the agent uses Policy Gradient Reward Design (PGRD) to fine-tune this internal reward based on actual performance.
- Incremental Learning: Once a task is finished, the agent uses the final, optimized internal reward as a training label to update its mapping function ($f$) via non-parametric kernel regression.
The agent uses UCT (Upper Confidence Bound applied to Trees) for planning, which is a standard but computationally expensive algorithm. By optimizing the internal reward, the authors essentially "cheat" the planning depth limit, providing the agent with a better "internal compass."
Experimental Evaluation
The approach was evaluated on two distinct simulated environments:
- Food-and-Shelter Domain: A 3x3 grid-world maze where the agent must balance eating food and repairing a shelter. The "tasks" change by varying the importance (reward/penalty) of food versus shelter.
- Network Routing Domain: A more complex, multi-agent environment where routers must move packets to minimize delay, loss, and power consumption. This domain was specifically modified to test changes in both the reward function and the transition dynamics (e.g., changing traffic patterns).
Key Insights and Results
- Rewards as a Locus for Transfer: Most transfer research focuses on knowledge (models) or behavior (policies). This paper argues that transferring the motivation (internal rewards) is a powerful and overlooked alternative, especially when the agent is computationally limited.
- Superior Performance: In the grid-world experiments, the Reward Mapping agent consistently outperformed "Sequential Transfer" (which just carries over the last task's reward) and "Non-transfer" agents across various task durations.
- Efficiency: The computational overhead of learning and applying the mapping function was found to be negligible compared to the cost of UCT planning. The "smart initialization" provided by the mapping function is essentially free performance.
- Robustness to Change: In the networking domain, the proposed method outperformed a state-of-the-art policy transfer method, particularly when both the rewards and the network dynamics changed simultaneously. By including environment parameters (like traffic probabilities) in the mapping function's input, the agent could transfer knowledge even when the "physics" of the world changed.
Limitations and Future Work
While the empirical results are strong, the authors note that more work is needed to understand the exact theoretical conditions under which reward mapping is guaranteed to outperform policy or value transfer. Additionally, the study used kernel regression for the mapping function; for much higher-dimensional task spaces, more sophisticated function approximators (like deep neural networks) might be required. Finally, the success of the transfer depends heavily on the quality of the "labels" generated by the online adaptation (PGRD). If the online adaptation fails to find a good internal reward during a task, the mapping function will learn from noisy or poor data.
Summary updated