Reward Mapping for Transfer in Long-Lived Agents

Xiaoxiao Guo, Satinder P. Singh, Richard L. LewisNeurIPS 2013 · OpenReview

Introduction

This paper introduces a novel transfer learning approach for "long-lived" agents that solve a sequence of different tasks. Instead of transferring policies or value functions as is traditional, the authors propose learning a Reward Mapping Function. This function translates a new task's objective into an optimized "internal" reward signal, helping computationally limited agents perform better right from the start of a new task.

The Challenge of Bounded, Long-Lived Agents

In reinforcement learning (RL), we usually assume the reward function provided by the environment is the best signal to guide an agent. However, for "bounded" agents—those with limited memory or planning time—the objective reward might be too sparse or misleading to lead to good behavior quickly.

The challenge is even greater for long-lived agents that face a sequence of different tasks. While traditional transfer learning focuses on reusing policies (how to act) or value functions (how good a state is), these methods often struggle when the environment's dynamics or the goals change significantly. The authors identify a critical gap: how can an agent transfer the "wisdom" of what kind of internal guidance worked best in the past to help it tackle a brand-new objective?

Methodology: Reward Mapping Transfer

The authors build on the Optimal Rewards Problem (ORP), which distinguishes between the objective reward (what the designer wants) and the internal reward (the signal actually used to guide the agent's planning).

Comparison of agent architectures across a sequence of tasks, highlighting the proposed Reward Mapping Transfer ORP Agent (c).

The core contribution is the Reward Mapping-Transfer ORP Agent. Its workflow operates as follows:

  1. The Mapping Function ($f$): The agent maintains a function that maps objective reward parameters ($\theta^o$) to initial internal reward parameters ($\theta^i$).
  2. Initialization: When a new task starts, the agent uses this mapping to "guess" a high-quality starting internal reward.
  3. Online Adaptation (PGRD): While performing the task, the agent uses Policy Gradient Reward Design (PGRD) to fine-tune this internal reward based on actual performance.
  4. Incremental Learning: Once a task is finished, the agent uses the final, optimized internal reward as a training label to update its mapping function ($f$) via non-parametric kernel regression.

The agent uses UCT (Upper Confidence Bound applied to Trees) for planning, which is a standard but computationally expensive algorithm. By optimizing the internal reward, the authors essentially "cheat" the planning depth limit, providing the agent with a better "internal compass."

Experimental Evaluation

The approach was evaluated on two distinct simulated environments:

  • Food-and-Shelter Domain: A 3x3 grid-world maze where the agent must balance eating food and repairing a shelter. The "tasks" change by varying the importance (reward/penalty) of food versus shelter.
  • Network Routing Domain: A more complex, multi-agent environment where routers must move packets to minimize delay, loss, and power consumption. This domain was specifically modified to test changes in both the reward function and the transition dynamics (e.g., changing traffic patterns).

Key Insights and Results

  • Rewards as a Locus for Transfer: Most transfer research focuses on knowledge (models) or behavior (policies). This paper argues that transferring the motivation (internal rewards) is a powerful and overlooked alternative, especially when the agent is computationally limited.
  • Superior Performance: In the grid-world experiments, the Reward Mapping agent consistently outperformed "Sequential Transfer" (which just carries over the last task's reward) and "Non-transfer" agents across various task durations.
  • Efficiency: The computational overhead of learning and applying the mapping function was found to be negligible compared to the cost of UCT planning. The "smart initialization" provided by the mapping function is essentially free performance.
  • Robustness to Change: In the networking domain, the proposed method outperformed a state-of-the-art policy transfer method, particularly when both the rewards and the network dynamics changed simultaneously. By including environment parameters (like traffic probabilities) in the mapping function's input, the agent could transfer knowledge even when the "physics" of the world changed.

Limitations and Future Work

While the empirical results are strong, the authors note that more work is needed to understand the exact theoretical conditions under which reward mapping is guaranteed to outperform policy or value transfer. Additionally, the study used kernel regression for the mapping function; for much higher-dimensional task spaces, more sophisticated function approximators (like deep neural networks) might be required. Finally, the success of the transfer depends heavily on the quality of the "labels" generated by the online adaptation (PGRD). If the online adaptation fails to find a good internal reward during a task, the mapping function will learn from noisy or poor data.

Summary updated

Abstract

We consider how to transfer knowledge from previous tasks to a current task in long-lived and bounded agents that must solve a sequence of MDPs over a finite lifetime. A novel aspect of our transfer approach is that we reuse reward functions. While this may seem counterintuitive, we build on the insight of recent work on the optimal rewards problem that guiding an agent's behavior with reward functions other than the task-specifying reward function can help overcome computational bounds of the agent. Specifically, we use good guidance reward functions learned on previous tasks in the sequence to incrementally train a reward mapping function that maps task-specifying reward functions into good initial guidance reward functions for subsequent tasks. We demonstrate that our approach can substantially improve the agent's performance relative to other approaches, including an approach that transfers policies.
BibTeX
@inproceedings{guo2013reward,
  author = {Xiaoxiao Guo and Satinder P. Singh and Richard L. Lewis},
  title = {{Reward Mapping for Transfer in Long-Lived Agents}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2013},
  url = {https://openreview.net/forum?id=B1WmY8ZOWS},
}
Figure