Abstract
Representation learning lies at the heart of the empirical success of deep learning for dealing with the curse of dimensionality. However, the power of representation learning has not been fully exploited yet in reinforcement learning (RL), due to i), the trade-off between expressiveness and tractability; and ii), the coupling between exploration and representation learning. In this paper, we first reveal the fact that under some noise assumption in the stochastic control model, we can obtain the linear spectral feature of its corresponding Markov transition operator in closed-form for free. Based on this observation, we propose Spectral Dynamics Embedding (SPEDE), which breaks the trade-off and completes optimistic exploration for representation learning by exploiting the structure of the noise. We provide rigorous theoretical analysis of SPEDE, and demonstrate the practical superior performance over the existing state-of-the-art empirical algorithms on several benchmarks.
Introduction
Reinforcement learning (RL) agents frequently encounter the "curse of dimensionality," where the sheer volume of potential states in real-world environments makes individual tracking impossible. Representation learning offers a solution by mapping complex states into lower-dimensional, manageable features. However, this introduces a significant challenge: the coupling of exploration and representation learning. Effective exploration requires a good model of the environment, yet learning such a model requires extensive exploration. Furthermore, a trade-off exists between model expressiveness and computational tractability; highly expressive deep neural networks often lead to optimization problems that are difficult to solve optimally.
Methodology: Spectral Dynamics Embedding (SPEDE)
The core innovation of this work is the realization that the inherent noise in stochastic control systems can be leveraged to derive representations. In many systems, the transition to a next state $s'$ is defined by a dynamics function $f^*(s, a)$ plus additive noise $\epsilon$.
The "Free Lunch" of Noise Structure
If the noise $\epsilon$ follows a known distribution, such as an isotropic Gaussian, the transition probability density can be decomposed into an inner product of two feature maps. This property, derived from Gaussian kernels or Random Fourier Transforms, allows for the derivation of a linear representation of the Markov transition operator "for free" once the dynamics function $f^*$ is estimated. This transformation converts a complex nonlinear control problem into a Linear MDP, significantly simplifying planning and exploration.
Algorithm Design
SPEDE employs Thompson Sampling for exploration, maintaining a posterior distribution over possible dynamics functions. By sampling a function in each episode and planning accordingly, the agent is naturally incentivized to visit uncertain regions. For practical implementation in continuous tasks, the framework utilizes Stochastic Gradient Langevin Dynamics (SGLD) to sample from the posterior, integrating the resulting representation into the Soft Actor-Critic (SAC) framework.
Experimental Evaluation
The performance of SPEDE was evaluated using the OpenAI MuJoCo control suite via the Model-Based Baseline Library (MBBL). The benchmarks included standard dynamics tasks like Swimmer and Reacher, as well as complex tasks with early termination (ET), such as Ant-ET, Hopper-ET, and Humanoid-ET.
Key Results
The proposed approach offers both theoretical guarantees and empirical advantages. Theoretically, the authors establish a sublinear regret bound, ensuring that the agent's performance converges toward the optimal strategy regardless of the state space size.
Empirically, SPEDE and its regularized variant, SPEDE-REG, demonstrate substantial improvements over state-of-the-art baselines. As shown in the results table below, SPEDE consistently outperforms model-based algorithms like PETS and ME-TRPO. Notably, in the challenging Humanoid-ET task, the standard SAC algorithm achieves a score of approximately 1794, whereas SPEDE-REG more than doubles this performance to approximately 3747.
Discussion and Future Work
While SPEDE provides a powerful bridge between theoretical RL and practical deep learning, it relies on specific noise distributions (e.g., Gaussian, Laplace). Environments with highly asymmetric or non-standard noise may require further adaptations. Additionally, while the practical implementation uses Random Fourier Features, increasing the feature count to improve accuracy involves a computational trade-off. Future research aims to extend rigorous regret bounds to the regularized versions of the algorithm that performed best in empirical tests.
Summary updated