Intrinsically Motivated Self-supervised Learning in Reinforcement Learning

Yue Zhao, Chenzhuang Du, Hang Zhao, Tiejun Li

This paper presents a clever, "why didn't we think of this sooner?" approach to improving how robots learn from visual information. By repurposing the "struggle" of a learning model as a reward signal, the authors significantly boost both the speed of learning and the ability of agents to handle messy, real-world environments.

TL;DR

The researchers introduce Intrinsically Motivated Self-supervised Learning (IM-SSR), a framework that treats the loss from self-supervised auxiliary tasks as an intrinsic reward for reinforcement learning agents. This simple "plug-and-play" modification encourages agents to explore novel states and build robust representations, leading to state-of-the-art performance in sparse-reward and visually complex robotics tasks.

What Problem Does This Paper Address?

Reinforcement Learning (RL) from pixels (like camera feeds) is notoriously difficult. To help, researchers often use Self-Supervised Learning (SSL) as an "auxiliary task." While the agent learns to move, a separate part of the model learns to understand the images (e.g., by predicting if two images are different views of the same scene).

However, there is a gap: the representation-learning part and the decision-making part are usually siloed. The agent's policy doesn't "know" how well the visual encoder is doing. This leads to two major issues:

  1. Inefficient Exploration: In environments where rewards are rare (sparse rewards), agents struggle to find anything useful to do.
  2. Poor Generalization: Agents often overfit to specific backgrounds or lighting, failing when moved from a simulator to a slightly different environment (the "nuisance" problem).

Methods

The core idea of IM-SSR is to bridge the gap between learning and acting.

In a standard setup, an SSL task generates a loss value ($L_{SSL}$) that tells you how much the model is struggling to represent a specific observation. IM-SSR takes this loss, normalizes it, and adds it to the environment's reward: $$R_{total} = R_{extrinsic} + \beta \cdot R_{intrinsic}$$ Where $R_{intrinsic}$ is essentially the SSL loss.

Why does this work? The authors provide a mathematical decomposition showing that SSL loss actually tracks two critical things:

  • Exploration Bonus: High loss often occurs in states the agent hasn't seen much (novelty).
  • Robustness Bonus: High loss also occurs in states that are "vulnerable" to visual noise or distortions.

By rewarding the agent for visiting states with high SSL loss, the agent is naturally "motivated" to explore new areas and spend more time in visually complex states until it masters them. Crucially, this requires nearly zero additional computational cost because the SSL loss is already being calculated.

Datasets

The authors evaluated their method using the DeepMind Control Suite (DMControl), a standard for robotics simulation. To test generalization, they used the DMControl-GB (Generalization Benchmark), which adds challenging visual "nuisances" like:

  • color_hard: Randomly changing the colors of the robot and environment.
  • video_hard: Replacing the background with dynamic, natural videos (e.g., driving through a city).

They tested these on various tasks including cartpole, finger, pendulum, reacher, and walker.

Key Insights and Ideas

  • The "Aha" Moment: The "error" in a self-supervised task isn't just a signal to update weights; it's a map of where the agent is most ignorant. Rewarding the agent for seeking out its own ignorance is a powerful heuristic for exploration.
  • No New Networks Needed: Unlike other exploration methods (like Random Network Distillation) that require extra "target" and "predictor" networks, IM-SSR uses the auxiliary task you were likely already using.
  • Sparse Reward Specialist: The method is particularly effective when the environment doesn't provide frequent feedback. The SSL loss acts as a "breadcrumb trail" of interestingness that keeps the agent engaged until it finds the real goal.

Key Results

  • Superior Sample Efficiency: On the cartpole_swingup_sparse task, IM-SSR achieved significantly higher rewards much earlier in training compared to baselines like CURL and SODA.
  • Robust Generalization: In the video_hard environment (the most difficult visual test), IM-SODA (the authors' method) maintained high performance while the standard SODA baseline's performance plummeted.
  • Stability: The researchers found that IM-SSR reduced the variance between different training runs, making the algorithms more reliable.

Limitations and Future Work

  • Hyperparameter Sensitivity: The weight of the intrinsic reward ($\beta$) is a new hyperparameter. While the authors used a simple decaying schedule, they note that an adaptive schedule could make the system even more robust.
  • Scope of Tasks: The experiments were limited to the DMControl suite. While these are standard, testing on higher-dimensional real-world robotics or more complex 3D environments (like navigation) would be the next logical step.
  • Interaction Complexity: The paper treats the SSL loss as a simple additive reward. Future work could explore more complex ways for the representation learner and the policy to communicate.

Summary updated

Abstract

In vision-based reinforcement learning (RL) tasks, it is prevalent to assign auxiliary tasks with a surrogate self-supervised loss so as to obtain more semantic representations and improve sample efficiency. However, abundant information in self-supervised auxiliary tasks has been disregarded, since the representation learning part and the decision-making part are separated. To sufficiently utilize information in auxiliary tasks, we present a simple yet effective idea to employ self-supervised loss as an intrinsic reward, called Intrinsically Motivated Self-Supervised learning in Reinforcement learning (IM-SSR). We formally show that the self-supervised loss can be decomposed as exploration for novel states and robustness improvement from nuisance elimination. IM-SSR can be effortlessly plugged into any reinforcement learning with self-supervised auxiliary objectives with nearly no additional cost. Combined with IM-SSR, the previous underlying algorithms achieve salient improvements on both sample efficiency and generalization in various vision-based robotics tasks from the DeepMind Control Suite, especially when the reward signal is sparse.
BibTeX
@inproceedings{zhao2022intrinsically,
  author = {Yue Zhao and Chenzhuang Du and Hang Zhao and Tiejun Li},
  title = {{Intrinsically Motivated Self-supervised Learning in Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=kI207HSfVQ2},
}

Loading related content...

Figure