Abstract
Reinforcement learning (RL) performance is highly sensitive to hyperparameters, often requiring computationally expensive outer-loop optimization. This paper proposes an online auto-tuning method for self-supervised robotic learning by leveraging the Evidence Lower Bound (ELBO) of a Variational Auto-Encoder (VAE). By demonstrating that the negative ELBO correlates with environmental data diversity, the authors automate the tuning of exploration steps, buffer size, and update frequency. The approach achieves state-of-the-art performance with significantly reduced computational overhead and memory usage compared to traditional tuning methods.
Introduction
Reinforcement Learning (RL) is notoriously sensitive to hyperparameter selection. Parameters such as exploration steps ($N_e$), buffer size ($N_b$), and update frequency ($N_\theta$) can dictate whether an agent successfully learns a task or fails entirely. Traditionally, finding optimal values requires an "outer loop" search—such as grid search or Bayesian optimization—which involves running the entire training process numerous times. This is particularly inefficient in robotics, where training time and hardware wear are significant constraints.
In self-supervised robotic learning frameworks like Reinforcement Learning with Imagined Goals (RIG), agents use VAEs to generate goals from visual observations. As the environment's complexity shifts—for instance, moving from simple reaching to multi-object manipulation—fixed hyperparameters often become suboptimal. This paper addresses the need for dynamic, online adaptation of these parameters.
Methodology: ELBO as a Diversity Metric
The core innovation of this work is the mathematical link between Information Theory and generative modeling. The authors propose replacing multiple sensitive hyperparameters with a single scaling factor, $\xi$, driven by the VAE's loss function.
- Diversity as Entropy: The diversity of a dataset can be quantified by its empirical entropy.
- Entropy and ELBO: The authors prove that the negative ELBO provides a lower bound for the diversity of observations.
- Dynamic Scaling: As the environment becomes more diverse (indicated by a rising negative ELBO), the agent requires more exploration to capture novel states, a larger buffer to store these experiences, and more gradient updates to process the information.
The system scales its learning effort dynamically: $N_e, N_\theta \propto -\text{ELBO}$ and $N_b \propto -\text{ELBO} \cdot \text{path length}$.
Experimental Evaluation
The approach was validated across a spectrum of tasks:
- Image Diversity: Using MNIST and Fashion-MNIST to confirm that ELBO tracks the number of unique classes in a dataset.
- Simulated Robotics: 2D Navigation and Visual Multi-Object Push tasks.
- Real-World Robotics: A "Real Reach" task using a physical Elfin 10 robotic arm equipped with a Kinect camera.
- Curriculum Learning: Scenarios where environment complexity (e.g., obstacles or camera angles) changed mid-training.
Key Results and Discussion
The auto-tuning method demonstrated several advantages:
- Efficiency: Compared to Optuna, a leading hyperparameter optimizer, this method achieved equivalent task performance while utilizing fewer gradient updates, fewer exploration steps, and less memory.
- Online Adaptation: Unlike traditional tuning, this process occurs within a single training run. The agent detects changes in data diversity via the ELBO and adjusts parameters immediately, allowing it to succeed in curriculum learning scenarios where fixed-parameter agents often fail.
- Reduced Search Space: By consolidating three sensitive parameters into one scaling factor ($\xi$), the method significantly simplifies the tuning process for researchers.
Conclusion
By utilizing the ELBO—a metric already computed in many self-supervised systems—this method provides "free" information for hyperparameter tuning. While the scaling factor $\xi$ still requires initial selection, it proves more robust than the original parameters. Future work may explore extending this "diversity meter" concept to other self-supervised losses, such as those used in contrastive learning, to broaden the applicability of online auto-tuning in RL.
Summary updated