Decoupled Learning of Environment Characteristics for Safe Exploration

Pieter Van Molle, Tim Verbelen, Steven Bohez, Sam Leroux, Pieter Simoens, Bart DhoedtarXiv 2017 · arXiv

Introduction

Traditional reinforcement learning (RL) typically relies on a monolithic reward signal to guide an agent's behavior. In such systems, the agent must simultaneously learn the fundamental rules of its environment (e.g., "avoiding obstacles") and the specific objectives of a task (e.g., "reaching a goal"). This lack of distinction leads to two primary challenges: poor transferability across different tasks within the same environment and high risk during the exploration phase. When an agent is tasked with a new objective, it often resets its entire knowledge base, leading to costly and potentially catastrophic failures as it re-explores dangerous states.

Methodology: Decoupling Reward and Value Functions

To address these issues, this paper proposes a decoupled approach that separates an agent’s learning into two distinct components: a survival function and a task function. By isolating environmental characteristics from task-specific goals, the agent can reuse its understanding of "safety" across multiple missions.

The Decoupled Architecture

The core innovation lies in the decomposition of the reward function $R$. Instead of a single signal, the authors define two components:

  1. Environmental Reward ($R_{\epsilon}(s)$): Represents survival-related feedback, such as penalties for collisions.
  2. Task-Specific Reward ($R_i(s)$): Represents goal-oriented feedback, such as bonuses for item collection.

This reward separation leads to a corresponding decomposition of the action-value function (Q-function): $$Q(s, a) = Q_{\epsilon}(s, a) + Q_i(s, a)$$

Figure 1: The decoupled architecture showing the separation of survival and task-specific reward signals and their integration into the action selection process.

Safe Exploration via Action Filtering

When transitioning to a new task, the agent retains the learned survival function ($Q_{\epsilon}$) while resetting the task function ($Q_i$). During exploration, the agent utilizes the survival network as a "safety rail." Rather than selecting actions purely at random (as in standard $\epsilon$-greedy exploration), the agent filters its choices, sampling only from a subset of actions deemed "safe" by the survival network. This significantly reduces the frequency of catastrophic failures during the learning process.

Experimental Setup

The proposed method was evaluated in a custom 11x11 Gridworld environment. The grid contains an agent, obstacles (red blocks with a 15% spawn probability), and two types of collectibles (yellow and blue blocks with a 5% spawn probability). Episodes were terminated after either a crash or 50 steps. The agent was first trained to collect one color while avoiding obstacles, then transitioned to a second task of collecting the other color.

Results and Discussion

The researchers compared three approaches: Naive Learning (starting from scratch for each task), Transfer Learning (reusing all weights from a previous task), and the proposed Decoupled Learning.

Survivability and Performance

The decoupled agent demonstrated superior performance during the training phase of the second task. As shown in the performance metrics, the decoupled approach maintained a significantly higher average return compared to both naive and standard transfer learning methods.

Figure 3: Average return per episode during the training of a new task. The decoupled approach (blue) achieves significantly higher returns by maintaining survival throughout the exploration phase.

This higher return is attributed to the agent's increased longevity; by avoiding obstacles from the start of the new task, the agent survives longer in each episode, allowing it to gather more relevant experience per attempt. In contrast, the naive agent failed to converge within 60,000 episodes, highlighting the difficulty of learning survival and task goals simultaneously in hostile environments.

Stability and Convergence

While the decoupled agent matched the convergence rate of standard transfer learning, it exhibited higher variance in action-value estimates. This suggests that while the separation of concerns improves safety and initial performance, the interaction between the two independent networks introduces additional complexity in stabilizing the final policy.

Conclusion

Decoupling survival from task-specific objectives provides a robust framework for safe exploration in RL. By treating environment-specific knowledge as a foundational skill that persists across tasks, agents can explore new goals with a built-in "sense of self-preservation." Future work will focus on automating the reward splitting process and testing the architecture in more complex, continuous environments such as real-world robotics.

Summary updated

Abstract

Reinforcement learning is a proven technique for an agent to learn a task. However, when learning a task using reinforcement learning, the agent cannot distinguish the characteristics of the environment from those of the task. This makes it harder to transfer skills between tasks in the same environment. Furthermore, this does not reduce risk when training for a new task. In this paper, we introduce an approach to decouple the environment characteristics from the task-specific ones, allowing an agent to develop a sense of survival. We evaluate our approach in an environment where an agent must learn a sequence of collection tasks, and show that decoupled learning allows for a safer utilization of prior knowledge.
BibTeX
@misc{molle2017decoupled,
  author = {Pieter Van Molle and Tim Verbelen and Steven Bohez and Sam Leroux and Pieter Simoens and Bart Dhoedt},
  title = {{Decoupled Learning of Environment Characteristics for Safe Exploration}},
  year = {2017},
  eprint = {1708.02838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.02838},
}
Figure