Introduction
Current Meta-Reinforcement Learning (Meta-RL) benchmarks often rely on synthetic, narrow tasks like maze navigation or simple locomotion. This paper introduces a new meta-HVAC benchmark that uses realistic building physics and exogenous variables—such as fluctuating energy prices and occupancy—to provide a more challenging and diverse environment for evaluating how well agents generalize to real-world control problems.
The Meta-HVAC Benchmark
Meta-RL aims to create agents that can "learn how to learn," adapting quickly to new tasks. However, real-world tasks involve exogenous stochasticity—random external factors like weather or market prices. Existing benchmarks lack this complexity, making it difficult to determine if Meta-RL algorithms are ready for industrial applications like smart building management.
The proposed benchmark frames HVAC control as a Reinforcement Learning problem where an agent manages thermostat set points to balance occupant comfort and energy costs. The simulation environment is defined by four variable factors that create a broad task distribution:
Methods
The authors evaluate $RL^2$, a standard meta-learning architecture that uses a Long Short-Term Memory (LSTM) network to maintain a "memory" of the current task's dynamics. They compare this meta-learner against ACKTR, a high-performance, task-specific RL algorithm that serves as a baseline for what a "perfectly adapted" agent should achieve. Training was conducted on a distribution of eight different building configurations (six 1st-order and two 2nd-order models).
Key Results and Insights
The benchmark highlights several critical aspects of Meta-RL performance:
- Physics and Objectives as Task Variables: By changing the underlying simulation model (e.g., from 1st-order to 2nd-order physics) or the price model (e.g., flat vs. real-time stochastic), the benchmark forces agents to adapt to different transition dynamics and fundamental objectives.
- Performance Parity: When the test task was within the training distribution, the $RL^2$ meta-learner achieved rewards nearly identical to the task-specific ACKTR baseline, demonstrating effective "zero-shot" adaptation.
- Distribution Sensitivity: Meta-RL agents are highly sensitive to the training distribution. As shown in Figure 1, an agent trained only on 1st-order models ($T_1$) performed significantly worse on 2nd-order models with stochastic pricing compared to an agent trained on a mix of both ($T_1, T_2$).
Furthermore, including more complex 2nd-order models in the training set allows the agent to reach a higher reward ceiling much faster when encountering a new, complex building.
Limitations and Future Work
A major hurdle in Meta-RL is the i.i.d. assumption, where training and test tasks are assumed to come from the same distribution. In practice, agents may encounter buildings with physics or price structures never seen before. Future research should focus on designing algorithms that do not rely on identical training and testing distributions and exploring "out-of-distribution" tasks, such as sudden grid failures.
Summary updated