Introduction
Reinforcement Learning (RL) agents frequently encounter challenges when operating in high-dimensional, noisy, or sparse observation spaces. A common strategy to mitigate these issues involves using Autoencoders (AEs) to compress raw environmental data into a lower-dimensional "latent space." However, static AE architectures often fail to balance bias and variance: small networks may underfit complex environments, while excessively large networks may overfit to noise.
This paper introduces the Self-evolving Autoencoder Embedded Q-Network (SAQN), a framework that autonomously adapts its neural architecture to environmental complexity. By employing a bias-variance regulatory strategy to grow or prune neurons, SAQN learns efficient latent representations that enable a Q-Network to solve complex tasks with greater speed and reliability than fixed-architecture models.
The SAQN Framework
The SAQN architecture consists of two primary components: a Self-evolving Autoencoder (SA) and a Q-Network (QN).
The process begins with a pre-training phase where the agent explores the environment randomly. During this phase, the SA monitors its reconstruction error using a mathematically rigorous bias-variance regulatory strategy based on a probit approximation.
- Growth: If the network detects high bias (underfitting), it adds a new hidden neuron to increase capacity.
- Pruning: If high variance (overfitting) is detected, the network removes the neuron contributing the least to the output.
Once the SA evolves a concise latent representation, these compressed states are fed into the Q-Network, which learns the optimal policy based on these refined features. This approach ensures the network topology is "just right" for the specific task at hand.
Experimental Evaluation
The authors evaluated SAQN against standard Q-Networks (QN) and fixed-architecture Integrated Autoencoders (AQN) across several benchmarks:
- Control Tasks: CartPole-v0 and LunarLander-v2.
- Sparse Reward Environments: Minigrid navigation.
- Real-world Application: Molecular optimization, involving a 2049-dimensional state space to "build" molecules with high drug-likeness (QED) scores.
Results and Discussion
SAQN consistently outperformed baseline models in both convergence speed and final performance. In the LunarLander environment, SAQN converged in an average of 276 episodes, nearly twice as fast as the standard Q-Network (548 episodes).
In the high-dimensional molecular optimization task, SAQN's adaptive complexity was particularly evident. The network evolved to 2052 neurons to capture the sparse nature of molecular strings, discovering 1,430 high-QED molecules—significantly more than the QN (302) or the fixed AQN (826). Visualization via t-SNE further confirmed that SAQN’s latent space provides superior state separation, effectively disentangling "good" states from "bad" ones and reducing overestimation bias.
Conclusion
The SAQN framework demonstrates that autonomous architectural evolution is a powerful tool for reinforcement learning in complex environments. While the self-evolving process introduces some computational overhead, the gains in learning efficiency and reliability are substantial. Future research will explore extending this logic to continuous action spaces by integrating self-evolving structures with Variational Autoencoders (VAEs) and Actor-Critic frameworks.
Summary updated