Utilizing Novelty-Based Evolution Strategies to Train Transformers in Reinforcement Learning

Matyás Lorenc, Roman Neruda

TL;DR

This paper explores using novelty-based evolution strategies (NS-ES and NSR-ES) to train complex transformer architectures for reinforcement learning, specifically Decision Transformers. While standard evolution strategies (OpenAI-ES) can train these models, the researchers investigate if novelty can provide sufficient training signals. They found that NSR-ES shows promise, achieving comparable results to OpenAI-ES with increased computation, but pretraining efforts to accelerate training were unsuccessful.

What Problem Does This Paper Address?

Reinforcement learning (RL) aims to train agents to make optimal decisions in an environment to maximize rewards. Traditional RL methods often rely on gradient-based approaches, which can struggle with complex, non-differentiable models or environments with sparse/deceptive rewards. Evolutionary algorithms, like evolution strategies (ES), offer a derivative-free alternative that can be highly parallelizable and exhibit better exploration.

Recently, transformer architectures, originally successful in natural language processing and computer vision, have been adapted for RL tasks, notably through models like the Decision Transformer. These models treat RL as a sequence modeling problem, leveraging the transformer's ability to process historical context. However, training these larger, more complex transformer models with RL algorithms, even derivative-free ones, presents a challenge. This paper investigates whether novelty-based search, which encourages exploration of diverse behaviors rather than solely optimizing for high rewards, can effectively train these advanced transformer architectures in RL.

Methods

The core of this research lies in applying and evaluating two novelty-based evolution strategies:

  • NS-ES (Novelty Search ES): This is a pure novelty search algorithm. Instead of optimizing for a fitness function (reward), it prioritizes exploring behaviors that are novel and different from those already encountered.
  • NSR-ES (Novelty Search with Reward ES): This is a quality-diversity algorithm. It combines both novelty and fitness (reward) to guide the search, aiming for both high-performing and diverse behaviors.

Both NS-ES and NSR-ES are built upon the OpenAI-ES algorithm, a representative of natural evolution strategies. OpenAI-ES models the agent's policy as a probability distribution (typically Gaussian) over its parameters. It generates offspring by sampling from this distribution, evaluates their performance, and updates the distribution's mean to improve expected future fitness.

The researchers also explored pretraining as a method to accelerate the training process. This involved training a simpler, feed-forward network first and then using its learned weights to initialize the transformer model before applying the novelty-based evolution strategies.

Datasets

The experiments were conducted in the MuJoCo Humanoid environment using the OpenAI Gym toolkit. This environment is described as one of the most complex and challenging standard environments within MuJoCo. The paper does not introduce new datasets or benchmarks.

Key Insights and Ideas

  • Novelty for Complex Models: While novelty search is generally useful for environments with sparse or deceptive rewards, this paper tests its efficacy on complex transformer architectures where the objective is clearer.
  • NSR-ES Shows Promise: The quality-diversity approach of NSR-ES proved more effective than pure novelty search (NS-ES) for training Decision Transformers, suggesting that a balance of novelty and objective is beneficial for these models.
  • Pretraining Doesn't Help (Here): Surprisingly, attempts to accelerate training by pretraining the Decision Transformer were unsuccessful. In some cases, pretraining even seemed to hinder performance, a stark contrast to findings with simpler feed-forward models.
  • Computational Cost: Novelty-based methods, especially for larger models, require significantly more computation (more iterations and larger populations) compared to objective-based methods like OpenAI-ES to achieve comparable results.
  • Behavioral Characterization is Crucial: The effectiveness of novelty search heavily relies on a well-defined "behavior characteristic" and a suitable distance metric. For the Humanoid environment, the agent's final position was used as the characteristic.

Key Results

  • NS-ES: Showed some progress but required substantially more iterations to yield interesting agents, especially for Decision Transformers.
  • NSR-ES: Demonstrated more capability in training Decision Transformers, with performance comparable to OpenAI-ES, albeit with a threefold increase in computation. It was sometimes even capable of further training pretrained models, though this did not offer overall benefits.
  • Pretraining: In experiments with both OpenAI-ES and novelty-based strategies, pretraining did not accelerate training and, in some cases, appeared to be detrimental.
  • Distance Traveled: Figure 3 shows that agents trained with NSR-ES achieved better average distances traveled from their starting positions compared to NS-ES, though still requiring more training time than objective-based methods.

Limitations and Future Work

  • Computational Demands: The primary limitation is the high computational cost associated with novelty-based methods, particularly for training larger models like Decision Transformers. NS-ES, in particular, would need "many more iterations" to yield interesting agents.
  • Pretraining Ineffectiveness: The failure of pretraining to accelerate training is a significant finding that warrants further investigation. The authors hypothesize that this might be due to the nature of novelty search or issues with how pretraining interacts with the novelty signal.
  • Future Work - Improved Pretraining/Seeding: A promising avenue for future work is to explore alternative methods for leveraging previous knowledge. Instead of direct pretraining, the authors suggest a "behavior cloning" approach where a new, larger metapopulation of models is trained by mimicking the behaviors of simpler, previously trained models, while retaining the behavior archive. This could prevent "reinventing the wheel" and potentially accelerate training.
  • Future Work - Hyperparameter Tuning: The authors also suggest that carefully tuning hyperparameters, especially those influencing the speed of training, might be crucial for overcoming the limitations observed with pretraining.

Summary updated

Abstract

In this paper, we experiment with novelty-based variants of OpenAI-ES, the NS-ES and NSR-ES algorithms, and evaluate their effectiveness in training complex, transformer-based architectures designed for the problem of reinforcement learning, such as Decision Transformers. We also test if we can accelerate the novelty-based training of these larger models by seeding the training with a pretrained models. The experimental results were mixed. NS-ES showed progress, but it would clearly need many more iterations for it to yield interesting agents. NSR-ES, on the other hand, proved quite capable of being straightforwardly used on larger models, since its performance appears as similar between the feed-forward model and Decision Transformer, as it was for the OpenAI-ES in our previous work.
BibTeX
@inproceedings{lorenc2025utilizing,
  author = {Matyás Lorenc and Roman Neruda},
  title = {{Utilizing Novelty-Based Evolution Strategies to Train Transformers in Reinforcement Learning}},
  booktitle = {ICTAI},
  year = {2025},
  doi = {10.1109/ictai66417.2025.00116},
}
Figure