Self-evolving Autoencoder Embedded Q-Network

J. Senthilnath, Bangjian Zhou, Zhen Wei Ng, Deeksha Aggarwal, Rajdeep Dutta, Ji Wei Yoon, Aye Phyu Phyu Aung, Keyu Wu, Min Wu, Xiaoli LiarXiv 2024 · DOI · arXiv

Introduction

Reinforcement Learning (RL) agents frequently encounter challenges when operating in high-dimensional, noisy, or sparse observation spaces. A common strategy to mitigate these issues involves using Autoencoders (AEs) to compress raw environmental data into a lower-dimensional "latent space." However, static AE architectures often fail to balance bias and variance: small networks may underfit complex environments, while excessively large networks may overfit to noise.

This paper introduces the Self-evolving Autoencoder Embedded Q-Network (SAQN), a framework that autonomously adapts its neural architecture to environmental complexity. By employing a bias-variance regulatory strategy to grow or prune neurons, SAQN learns efficient latent representations that enable a Q-Network to solve complex tasks with greater speed and reliability than fixed-architecture models.

The SAQN Framework

The SAQN architecture consists of two primary components: a Self-evolving Autoencoder (SA) and a Q-Network (QN).

The process begins with a pre-training phase where the agent explores the environment randomly. During this phase, the SA monitors its reconstruction error using a mathematically rigorous bias-variance regulatory strategy based on a probit approximation.

  • Growth: If the network detects high bias (underfitting), it adds a new hidden neuron to increase capacity.
  • Pruning: If high variance (overfitting) is detected, the network removes the neuron contributing the least to the output.

Once the SA evolves a concise latent representation, these compressed states are fed into the Q-Network, which learns the optimal policy based on these refined features. This approach ensures the network topology is "just right" for the specific task at hand.

Experimental Evaluation

The authors evaluated SAQN against standard Q-Networks (QN) and fixed-architecture Integrated Autoencoders (AQN) across several benchmarks:

  • Control Tasks: CartPole-v0 and LunarLander-v2.
  • Sparse Reward Environments: Minigrid navigation.
  • Real-world Application: Molecular optimization, involving a 2049-dimensional state space to "build" molecules with high drug-likeness (QED) scores.

Results and Discussion

SAQN consistently outperformed baseline models in both convergence speed and final performance. In the LunarLander environment, SAQN converged in an average of 276 episodes, nearly twice as fast as the standard Q-Network (548 episodes).

Performance comparison in the CartPole environment: SAQN (red) demonstrates significantly faster convergence compared to standard QN (blue) and fixed-architecture AQN (green).

In the high-dimensional molecular optimization task, SAQN's adaptive complexity was particularly evident. The network evolved to 2052 neurons to capture the sparse nature of molecular strings, discovering 1,430 high-QED molecules—significantly more than the QN (302) or the fixed AQN (826). Visualization via t-SNE further confirmed that SAQN’s latent space provides superior state separation, effectively disentangling "good" states from "bad" ones and reducing overestimation bias.

Conclusion

The SAQN framework demonstrates that autonomous architectural evolution is a powerful tool for reinforcement learning in complex environments. While the self-evolving process introduces some computational overhead, the gains in learning efficiency and reliability are substantial. Future research will explore extending this logic to continuous action spaces by integrating self-evolving structures with Variational Autoencoders (VAEs) and Actor-Critic frameworks.

Summary updated

Abstract

In the realm of sequential decision-making tasks, the exploration capability of a reinforcement learning (RL) agent is paramount for achieving high rewards through interactions with the environment. To enhance this crucial ability, we propose SAQN, a novel approach wherein a self-evolving autoencoder (SA) is embedded with a Q-Network (QN). In SAQN, the self-evolving autoencoder architecture adapts and evolves as the agent explores the environment. This evolution enables the autoencoder to capture a diverse range of raw observations and represent them effectively in its latent space. By leveraging the disentangled states extracted from the encoder generated latent space, the QN is trained to determine optimal actions that improve rewards. During the evolution of the autoencoder architecture, a bias-variance regulatory strategy is employed to elicit the optimal response from the RL agent. This strategy involves two key components: (i) fostering the growth of nodes to retain previously acquired knowledge, ensuring a rich representation of the environment, and (ii) pruning the least contributing nodes to maintain a more manageable and tractable latent space. Extensive experimental evaluations conducted on three distinct benchmark environments and a real-world molecular environment demonstrate that the proposed SAQN significantly outperforms state-of-the-art counterparts. The results highlight the effectiveness of the self-evolving autoencoder and its collaboration with the Q-Network in tackling sequential decision-making tasks.
BibTeX
@misc{senthilnath2024self,
  author = {J. Senthilnath and Bangjian Zhou and Zhen Wei Ng and Deeksha Aggarwal and Rajdeep Dutta and Ji Wei Yoon and Aye Phyu Phyu Aung and Keyu Wu and Min Wu and Xiaoli Li},
  title = {{Self-evolving Autoencoder Embedded Q-Network}},
  year = {2024},
  eprint = {2402.11604},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.11604},
}
Figure