Introduction
Traditional reinforcement learning (RL) typically relies on a monolithic reward signal to guide an agent's behavior. In such systems, the agent must simultaneously learn the fundamental rules of its environment (e.g., "avoiding obstacles") and the specific objectives of a task (e.g., "reaching a goal"). This lack of distinction leads to two primary challenges: poor transferability across different tasks within the same environment and high risk during the exploration phase. When an agent is tasked with a new objective, it often resets its entire knowledge base, leading to costly and potentially catastrophic failures as it re-explores dangerous states.
Methodology: Decoupling Reward and Value Functions
To address these issues, this paper proposes a decoupled approach that separates an agent’s learning into two distinct components: a survival function and a task function. By isolating environmental characteristics from task-specific goals, the agent can reuse its understanding of "safety" across multiple missions.
The Decoupled Architecture
The core innovation lies in the decomposition of the reward function $R$. Instead of a single signal, the authors define two components:
- Environmental Reward ($R_{\epsilon}(s)$): Represents survival-related feedback, such as penalties for collisions.
- Task-Specific Reward ($R_i(s)$): Represents goal-oriented feedback, such as bonuses for item collection.
This reward separation leads to a corresponding decomposition of the action-value function (Q-function): $$Q(s, a) = Q_{\epsilon}(s, a) + Q_i(s, a)$$
Safe Exploration via Action Filtering
When transitioning to a new task, the agent retains the learned survival function ($Q_{\epsilon}$) while resetting the task function ($Q_i$). During exploration, the agent utilizes the survival network as a "safety rail." Rather than selecting actions purely at random (as in standard $\epsilon$-greedy exploration), the agent filters its choices, sampling only from a subset of actions deemed "safe" by the survival network. This significantly reduces the frequency of catastrophic failures during the learning process.
Experimental Setup
The proposed method was evaluated in a custom 11x11 Gridworld environment. The grid contains an agent, obstacles (red blocks with a 15% spawn probability), and two types of collectibles (yellow and blue blocks with a 5% spawn probability). Episodes were terminated after either a crash or 50 steps. The agent was first trained to collect one color while avoiding obstacles, then transitioned to a second task of collecting the other color.
Results and Discussion
The researchers compared three approaches: Naive Learning (starting from scratch for each task), Transfer Learning (reusing all weights from a previous task), and the proposed Decoupled Learning.
Survivability and Performance
The decoupled agent demonstrated superior performance during the training phase of the second task. As shown in the performance metrics, the decoupled approach maintained a significantly higher average return compared to both naive and standard transfer learning methods.
This higher return is attributed to the agent's increased longevity; by avoiding obstacles from the start of the new task, the agent survives longer in each episode, allowing it to gather more relevant experience per attempt. In contrast, the naive agent failed to converge within 60,000 episodes, highlighting the difficulty of learning survival and task goals simultaneously in hostile environments.
Stability and Convergence
While the decoupled agent matched the convergence rate of standard transfer learning, it exhibited higher variance in action-value estimates. This suggests that while the separation of concerns improves safety and initial performance, the interaction between the two independent networks introduces additional complexity in stabilizing the final policy.
Conclusion
Decoupling survival from task-specific objectives provides a robust framework for safe exploration in RL. By treating environment-specific knowledge as a foundational skill that persists across tasks, agents can explore new goals with a built-in "sense of self-preservation." Future work will focus on automating the reward splitting process and testing the architecture in more complex, continuous environments such as real-world robotics.
Summary updated