MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator

Xiao-Yin Liu, Xiao-Hu Zhou, Guotao Li, Hao Li, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, Zeng-Guang HouIJCAI 2024

Introduction

Offline Reinforcement Learning (RL) offers a promising path for training agents using pre-collected datasets, avoiding the risks and costs associated with online exploration. However, the primary challenge in this domain is distribution shift. When an agent deviates from the behavior patterns present in the offline data, it often encounters "Out-of-Distribution" (OOD) states where its value estimates become unreliable. While model-based methods attempt to mitigate this by learning a simulator of the environment, these simulators are prone to inaccuracies, which agents can inadvertently exploit.

The Challenge of Offline RL

Current offline RL approaches struggle with two main issues: the distribution shift and the model gap. Model-based methods that rely on learned dynamics often suffer when the simulator "hallucinates" transitions. Previous attempts to address this, such as RAMBO or ARMOR, utilize adversarial training to create pessimistic models. While effective, these methods are computationally intensive and often lack robustness when faced with physical perturbations in the real-world environment.

The MICRO Framework

Presented at IJCAI-24, MICRO (Model-based offline RL with a Conservative Bellman Operator) introduces a more efficient and robust framework. Instead of training separate adversarial models, MICRO embeds pessimism directly into the learning process via a novel Conservative Bellman Operator.

Overview of the MICRO framework, integrating model-based and robust RL through a conservative Bellman operator.

The core components of MICRO include:

  • Pessimism by Design: The agent evaluates actions by considering the worst-case outcome within an "uncertainty set" of possible future states, rather than relying on a single average prediction.
  • Dual Reformulation: By applying Lagrange dual reformulation, the authors transform a complex optimization problem into a manageable task of finding the minimal Q-value within a local neighborhood.
  • Dynamic Penalty: A penalty term, $f(s, a)$, is derived to quantify the gap between expected and worst-case rewards, effectively tempering the agent's optimism in OOD regions.
  • Ensemble Dynamics: Uncertainty is estimated using an ensemble of neural networks. Greater disagreement among the networks leads to a larger uncertainty set and more cautious behavior.

Experimental Results

The researchers evaluated MICRO using the standard D4RL benchmark, focusing on the HalfCheetah, Hopper, and Walker2d environments across Medium, Medium-Replay, and Medium-Expert datasets.

Performance and Efficiency

MICRO demonstrated state-of-the-art performance, achieving an average normalized score of 97.1, surpassing both model-based and model-free baselines. Notably, it achieved these results with significantly lower computational overhead—nearly 3x faster than previous adversarial methods like RAMBO.

Performance comparison of MICRO against model-based and model-free offline RL baselines on D4RL benchmarks.

Robustness and Resilience

Beyond standard benchmarks, MICRO showed remarkable resilience to environmental changes. In tests involving variations in gravity (0.5x to 5x) and friction (0.5x to 1.5x), MICRO maintained stable performance while competing algorithms saw sharp declines. This suggests that the internal robustness to model errors translates effectively to external physical disturbances.

Conclusion

MICRO represents a significant step forward in making offline RL both practical and robust. By integrating pessimism into the Bellman operator, it achieves superior performance and environmental resilience without the prohibitive computational costs of adversarial training. Future research will focus on balancing this conservatism to handle more complex adversarial attacks without sacrificing performance.

Summary updated

BibTeX
@inproceedings{liu2024micro,
  author = {Xiao-Yin Liu and Xiao-Hu Zhou and Guotao Li and Hao Li and Mei-Jiang Gui and Tian-Yu Xiang and De-Xing Huang and Zeng-Guang Hou},
  title = {{MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure