Hyperparameter Auto-tuning in Self-Supervised Robotic Learning

Jiancong Huang, Juan Rojas, Matthieu Zimmer, Hongmin Wu, Yisheng Guan, Paul Weng

Abstract

Reinforcement learning (RL) performance is highly sensitive to hyperparameters, often requiring computationally expensive outer-loop optimization. This paper proposes an online auto-tuning method for self-supervised robotic learning by leveraging the Evidence Lower Bound (ELBO) of a Variational Auto-Encoder (VAE). By demonstrating that the negative ELBO correlates with environmental data diversity, the authors automate the tuning of exploration steps, buffer size, and update frequency. The approach achieves state-of-the-art performance with significantly reduced computational overhead and memory usage compared to traditional tuning methods.

Introduction

Reinforcement Learning (RL) is notoriously sensitive to hyperparameter selection. Parameters such as exploration steps ($N_e$), buffer size ($N_b$), and update frequency ($N_\theta$) can dictate whether an agent successfully learns a task or fails entirely. Traditionally, finding optimal values requires an "outer loop" search—such as grid search or Bayesian optimization—which involves running the entire training process numerous times. This is particularly inefficient in robotics, where training time and hardware wear are significant constraints.

In self-supervised robotic learning frameworks like Reinforcement Learning with Imagined Goals (RIG), agents use VAEs to generate goals from visual observations. As the environment's complexity shifts—for instance, moving from simple reaching to multi-object manipulation—fixed hyperparameters often become suboptimal. This paper addresses the need for dynamic, online adaptation of these parameters.

Methodology: ELBO as a Diversity Metric

The core innovation of this work is the mathematical link between Information Theory and generative modeling. The authors propose replacing multiple sensitive hyperparameters with a single scaling factor, $\xi$, driven by the VAE's loss function.

  1. Diversity as Entropy: The diversity of a dataset can be quantified by its empirical entropy.
  2. Entropy and ELBO: The authors prove that the negative ELBO provides a lower bound for the diversity of observations.
  3. Dynamic Scaling: As the environment becomes more diverse (indicated by a rising negative ELBO), the agent requires more exploration to capture novel states, a larger buffer to store these experiences, and more gradient updates to process the information.

The system scales its learning effort dynamically: $N_e, N_\theta \propto -\text{ELBO}$ and $N_b \propto -\text{ELBO} \cdot \text{path length}$.

The negative ELBO and its components (Reconstruction Loss and KL Divergence) tracked over training epochs, serving as a proxy for data diversity.

Experimental Evaluation

The approach was validated across a spectrum of tasks:

  • Image Diversity: Using MNIST and Fashion-MNIST to confirm that ELBO tracks the number of unique classes in a dataset.
  • Simulated Robotics: 2D Navigation and Visual Multi-Object Push tasks.
  • Real-World Robotics: A "Real Reach" task using a physical Elfin 10 robotic arm equipped with a Kinect camera.
  • Curriculum Learning: Scenarios where environment complexity (e.g., obstacles or camera angles) changed mid-training.
The real-world experimental setup featuring an Elfin 10 robot arm and a Kinect camera for capturing RGB observations.

Key Results and Discussion

The auto-tuning method demonstrated several advantages:

  • Efficiency: Compared to Optuna, a leading hyperparameter optimizer, this method achieved equivalent task performance while utilizing fewer gradient updates, fewer exploration steps, and less memory.
  • Online Adaptation: Unlike traditional tuning, this process occurs within a single training run. The agent detects changes in data diversity via the ELBO and adjusts parameters immediately, allowing it to succeed in curriculum learning scenarios where fixed-parameter agents often fail.
  • Reduced Search Space: By consolidating three sensitive parameters into one scaling factor ($\xi$), the method significantly simplifies the tuning process for researchers.

Conclusion

By utilizing the ELBO—a metric already computed in many self-supervised systems—this method provides "free" information for hyperparameter tuning. While the scaling factor $\xi$ still requires initial selection, it proves more robust than the original parameters. Future work may explore extending this "diversity meter" concept to other self-supervised losses, such as those used in contrastive learning, to broaden the applicability of online auto-tuning in RL.

Summary updated

Abstract

Policy optimization in reinforcement learning requires the selection of numerous hyperparameters across different environments. Fixing them incorrectly may negatively impact optimization performance leading notably to insufficient or redundant learning. Insufficient learning (due to convergence to local optima) results in under-performing policies whilst redundant learning wastes time and resources. The effects are further exacerbated when using single policies to solve multi-task learning problems. Observing that the Evidence Lower Bound (ELBO) used in Variational Auto-Encoders correlates with the diversity of image samples, we propose an auto-tuning technique based on the ELBO for self-supervised reinforcement learning. Our approach can auto-tune three hyperparameters: the replay buffer size, the number of policy gradient updates during each epoch, and the number of exploration steps during each epoch. We use a state-of-the-art self-supervised robot learning framework (Reinforcement Learning with Imagined Goals (RIG) using Soft Actor-Critic) as baseline for experimental verification. Experiments show that our method can auto-tune online and yields the best performance at a fraction of the time and computational resources. Code, video, and appendix for simulated and real-robot experiments can be found at the project page \url{www.JuanRojas.net/autotune}.
BibTeX
@article{huang2020hyperparameter,
  author = {Jiancong Huang and Juan Rojas and Matthieu Zimmer and Hongmin Wu and Yisheng Guan and Paul Weng},
  title = {{Hyperparameter Auto-tuning in Self-Supervised Robotic Learning}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2020},
  doi = {10.1109/lra.2021.3064509},
}
Figure