Abstract
Recent vision-based reinforcement learning (RL) methods have found extracting high-level features from raw pixels with self-supervised learning to be effective in learning policies. However, these methods focus on learning global representations of images, and disregard local spatial structures present in the consecutively stacked frames. In this paper, we propose a novel approach, termed self-supervised Paired Similarity Representation Learning (PSRL) for effectively encoding spatial structures in an unsupervised manner. Given the input frames, the latent volumes are first generated individually using an encoder, and they are used to capture the variance in terms of local spatial structures, i.e., correspondence maps among multiple frames. This enables for providing plenty of fine-grained samples for training the encoder of deep RL. We further attempt to learn the global semantic representations in the action aware transform module that predicts future state representations using action vectors as a medium. The proposed method imposes similarity constraints on the three latent volumes; transformed query representations by estimated pixel-wise correspondence, predicted query representations from the action aware transform model, and target representations of future state, guiding action aware transform with locality-inherent volume. Experimental results on complex tasks in Atari Games and DeepMind Control Suite demonstrate that the RL methods are significantly boosted by the proposed self-supervised learning of paired similarity representations.
Introduction
Training reinforcement learning (RL) agents directly from raw visual input is a significant challenge due to the high dimensionality and sample inefficiency of pixel-based learning. While contemporary self-supervised learning (SSL) techniques have improved performance by extracting high-level "global" features, they often overlook the rich information embedded in local spatial structures—specifically, how objects and textures deform between consecutive frames.
The authors of PSRL argue that these local deformations provide a valuable source of "free" supervision. By capturing fine-grained movement, an agent can achieve more precise control. The core challenge is integrating these local details with the global semantic understanding required to predict the outcomes of an agent's actions.
Methodology: Paired Similarity Representation Learning
PSRL introduces a "Local-guided Global" framework that aligns local spatial information with global action-driven predictions. The architecture consists of several key modules:
- Self-supervised Correspondence Estimation: The model learns to predict pixel-wise correspondence maps (analogous to optical flow) between consecutive frames without manual labels.
- Correspondence Aware Transform (CAT): This module uses the predicted correspondence maps to perform an "inverse warping," transforming current latent features into a "locally-inherent" representation of the future state.
- Action Aware Transform (AAT): This module predicts the future state representation by processing the current state and the agent's action. This captures global semantic changes driven by the agent's behavior.
- Paired Similarity Constraint: The central innovation of PSRL is a similarity loss that aligns three latent volumes: the locally-warped query representation, the action-predicted query representation, and the actual target representation of the future state.
By sharing prediction and projection heads across these modules, the local spatial information directly guides the learning of the global state representation, resulting in more robust features for the RL policy.
Experimental Evaluation
The proposed method was evaluated on two high-complexity benchmarks: Atari 2600 Games (26 environments) and the DeepMind Control Suite (DMControl) (6 continuous control tasks).
Atari 2600 Results
In the Atari benchmark, evaluated at 100K interaction steps to measure sample efficiency, PSRL demonstrated superior performance, achieving the highest scores in 13 out of 26 games. It significantly outperformed established baselines such as CURL and SPR, particularly in games with complex spatial dynamics like Alien and Seaquest.
DeepMind Control Suite Results
On DMControl, PSRL maintained its lead, achieving the highest mean scores in 4 out of 6 tasks at the 500K step mark (e.g., Finger Spin, Walker Walk). Along with RAD and DrQ, PSRL was among the fastest methods to converge, reaching high performance levels significantly earlier than standard pixel-based Soft Actor-Critic (SAC).
Discussion and Limitations
The "local-guided global" approach proves that spatial deformation is a powerful signal for training RL encoders. By using actions as a medium for prediction, PSRL learns representations that are functionally relevant to the task.
However, the method does have trade-offs. The additional modules increase training time (e.g., 16 hours for 500K steps on DMControl vs. 10–13 hours for CURL). Crucially, since these modules are only used during training, the inference time remains identical to simpler methods. Furthermore, PSRL's advantage diminishes in environments with sparse spatial structures, such as Pong, where correspondence matching provides less benefit.
Conclusion
PSRL presents a novel framework for visual RL that effectively bridges the gap between local spatial structures and global semantic predictions. By enforcing paired similarity constraints, it achieves state-of-the-art sample efficiency across diverse and complex environments. Future work may explore applying this philosophy to broader self-supervised learning domains like autonomous navigation and video representation.
Summary updated