Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning

Matyáš Lorenc, Roman Neruda

Abstract

In this paper, we experiment with novelty-based variants of OpenAI-ES, the NS-ES and NSR-ES algorithms, and evaluate their effectiveness in training complex, transformer-based architectures designed for the problem of reinforcement learning, such as Decision Transformers. We also test if we can accelerate the novelty-based training of these larger models by seeding the training with a pretrained models. The experimental results were mixed. NS-ES showed progress, but it would clearly need many more iterations for it to yield interesting agents. NSR-ES, on the other hand, proved quite capable of being straightforwardly used on larger models, since its performance appears as similar between the feed-forward model and Decision Transformer, as it was for the OpenAI-ES in our previous work.
BibTeX
@inproceedings{lorenc2025utilizing,
  author = {Matyáš Lorenc and Roman Neruda},
  title = {{Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning}},
  booktitle = {ICTAI},
  year = {2025},
  doi = {10.1109/ictai66417.2025.00116},
}

Loading related content...

Figure