Abstract
In vision-based reinforcement learning (RL) tasks, it is prevalent to assign auxiliary tasks with a surrogate self-supervised loss so as to obtain more semantic representations and improve sample efficiency. However, abundant information in self-supervised auxiliary tasks has been disregarded, since the representation learning part and the decision-making part are separated. To sufficiently utilize information in auxiliary tasks, we present a simple yet effective idea to employ self-supervised loss as an intrinsic reward, called Intrinsically Motivated Self-Supervised learning in Reinforcement learning (IM-SSR). We formally show that the self-supervised loss can be decomposed as exploration for novel states and robustness improvement from nuisance elimination. IM-SSR can be effortlessly plugged into any reinforcement learning with self-supervised auxiliary objectives with nearly no additional cost. Combined with IM-SSR, the previous underlying algorithms achieve salient improvements on both sample efficiency and generalization in various vision-based robotics tasks from the DeepMind Control Suite, especially when the reward signal is sparse.
Introduction
Reinforcement Learning (RL) from high-dimensional visual inputs, such as camera feeds, is notoriously difficult. To mitigate this, researchers often employ Self-Supervised Learning (SSL) as an auxiliary task to help the agent learn meaningful representations. However, a significant gap remains: the representation-learning component and the decision-making policy are typically siloed. The agent's policy does not "know" how well the visual encoder is performing, leading to two major issues: inefficient exploration in sparse-reward environments and poor generalization when faced with visual "nuisances" like changing backgrounds or lighting.
Methodology: The IM-SSR Framework
The core innovation of IM-SSR is bridging the gap between learning and acting by repurposing the SSL loss as an intrinsic reward signal. In a standard setup, an SSL task generates a loss value ($L_{SSL}$) that quantifies how much the model struggles to represent a specific observation. IM-SSR takes this loss, normalizes it, and adds it to the environment's extrinsic reward:
$$R_{total} = R_{extrinsic} + \beta \cdot R_{intrinsic}$$
This approach works because SSL loss naturally tracks two critical factors. First, it acts as an exploration bonus, as high loss often occurs in novel states the agent has rarely visited. Second, it serves as a robustness bonus, as high loss also occurs in states that are vulnerable to visual noise. By rewarding the agent for visiting states with high SSL loss, the agent is naturally motivated to explore new areas and spend more time in visually complex states until it masters them.
Experimental Setup and Results
The authors evaluated IM-SSR using the DeepMind Control Suite (DMControl). To test generalization, they used the DMControl-GB (Generalization Benchmark), which adds challenging visual nuisances such as video_hard, where backgrounds are replaced with dynamic, natural videos.
Key Findings
- Superior Sample Efficiency: On sparse-reward tasks like
cartpole_swingup_sparse, IM-SSR achieved significantly higher rewards much earlier in training compared to baselines like CURL and SODA. - Robust Generalization: In the
video_hardenvironment, IM-SSR maintained high performance while standard baselines plummeted, demonstrating its ability to ignore irrelevant visual noise. - Stability: The method reduced variance between different training runs, making the underlying algorithms more reliable.
Conclusion and Future Work
IM-SSR is a simple yet powerful "plug-and-play" modification that requires nearly zero additional computational cost, as the SSL loss is already calculated during training. By treating the "error" in a self-supervised task as a map of ignorance, the agent can effectively seek out and master complex environments. Future research may explore adaptive schedules for the intrinsic reward weight ($\beta$) and evaluate the framework in higher-dimensional 3D or real-world robotics settings.
Summary updated