Evaluating Meta-Reinforcement Learning through a HVAC Control Benchmark (Student Abstract)

Yashvir S. Grewal, Frits de Nijs, Sarah GoodwinAAAI 2021 · DOI

Introduction

Current Meta-Reinforcement Learning (Meta-RL) benchmarks often rely on synthetic, narrow tasks like maze navigation or simple locomotion. This paper introduces a new meta-HVAC benchmark that uses realistic building physics and exogenous variables—such as fluctuating energy prices and occupancy—to provide a more challenging and diverse environment for evaluating how well agents generalize to real-world control problems.

The Meta-HVAC Benchmark

Meta-RL aims to create agents that can "learn how to learn," adapting quickly to new tasks. However, real-world tasks involve exogenous stochasticity—random external factors like weather or market prices. Existing benchmarks lack this complexity, making it difficult to determine if Meta-RL algorithms are ready for industrial applications like smart building management.

The proposed benchmark frames HVAC control as a Reinforcement Learning problem where an agent manages thermostat set points to balance occupant comfort and energy costs. The simulation environment is defined by four variable factors that create a broad task distribution:

Table 1: Variable factors defining the meta-HVAC task distribution, including model complexity, price structures, occupancy, and grid limits.

Methods

The authors evaluate $RL^2$, a standard meta-learning architecture that uses a Long Short-Term Memory (LSTM) network to maintain a "memory" of the current task's dynamics. They compare this meta-learner against ACKTR, a high-performance, task-specific RL algorithm that serves as a baseline for what a "perfectly adapted" agent should achieve. Training was conducted on a distribution of eight different building configurations (six 1st-order and two 2nd-order models).

Key Results and Insights

The benchmark highlights several critical aspects of Meta-RL performance:

  • Physics and Objectives as Task Variables: By changing the underlying simulation model (e.g., from 1st-order to 2nd-order physics) or the price model (e.g., flat vs. real-time stochastic), the benchmark forces agents to adapt to different transition dynamics and fundamental objectives.
  • Performance Parity: When the test task was within the training distribution, the $RL^2$ meta-learner achieved rewards nearly identical to the task-specific ACKTR baseline, demonstrating effective "zero-shot" adaptation.
  • Distribution Sensitivity: Meta-RL agents are highly sensitive to the training distribution. As shown in Figure 1, an agent trained only on 1st-order models ($T_1$) performed significantly worse on 2nd-order models with stochastic pricing compared to an agent trained on a mix of both ($T_1, T_2$).
Figure 1: Comparison of reward performance across different model complexities and pricing structures for ACKTR and RL² variants.

Furthermore, including more complex 2nd-order models in the training set allows the agent to reach a higher reward ceiling much faster when encountering a new, complex building.

Limitations and Future Work

A major hurdle in Meta-RL is the i.i.d. assumption, where training and test tasks are assumed to come from the same distribution. In practice, agents may encounter buildings with physics or price structures never seen before. Future research should focus on designing algorithms that do not rely on identical training and testing distributions and exploring "out-of-distribution" tasks, such as sudden grid failures.

Summary updated

BibTeX
@inproceedings{grewal2021evaluating,
  author = {Yashvir S. Grewal and Frits de Nijs and Sarah Goodwin},
  title = {{Evaluating Meta-Reinforcement Learning through a HVAC Control Benchmark (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17889},
}
Figure