Abstract
While electric vehicles (EVs) continue to draw more attention as an alternative to traditional fossil fuel vehicles, the relatively short driving range of EVs remains a significant drawback. One of the most energy-intensive systems in EVs is the heating, ventilation, and air conditioning (HVAC) system. Most current HVAC systems utilize On/Off or PID control for actuators, but these methods often suffer from low efficiency and are difficult to apply in multiple-input multiple-output (MIMO) systems. This paper proposes a novel multi-agent deep reinforcement learning (MADRL) method to efficiently control low-level actuators. By providing independent rewards for each actuator agent, the system simultaneously optimizes for setpoint temperature, subcooling, and efficiency. Evaluated via a high-fidelity vehicle simulator, the MADRL-based method consumes only 53% of the energy of PID control on average during the transient phase.
Introduction
In internal combustion engine vehicles, cabin heating is often a byproduct of "waste heat." However, in an Electric Vehicle, every joule used for thermal management—whether heating or cooling—comes directly from the battery, significantly impacting the driving range.
Traditional HVAC control relies on PID (Proportional-Integral-Derivative) controllers. While robust, PID controllers are essentially reactive; they respond to current errors rather than planning an energy-efficient trajectory. Furthermore, they struggle with MIMO systems where actuators, such as the compressor and the electric expansion valve (EXV), exhibit complex interdependencies. This research aims to replace these reactive heuristics with an intelligent, predictive MADRL scheme that minimizes energy expenditure without sacrificing passenger comfort.
Methodology
The proposed framework utilizes Multi-Agent Reinforcement Learning (MARL) based on the Soft Actor-Critic (SAC) algorithm. The system architecture is built around two specialized agents: one for the Compressor (controlling RPM) and one for the Electric Expansion Valve (EXV). This decomposition prevents the high-power compressor from dominating the learning process, allowing the EXV agent to find subtle efficiency gains.
To optimize performance across different operational states, the researchers implemented phase-specific rewards. The system switches between a Transient Agent, focused on reaching the target temperature rapidly and efficiently, and a Steady Agent, focused on maintaining that temperature while optimizing the refrigeration cycle's efficiency.
Simulation and Training
The study utilized GT-SUITE, an industry-standard commercial simulation software, for vehicle thermal modeling. Training involved 500 episodes of interaction per setpoint. Testing was conducted across target temperatures ranging from 4°C to 12°C, with a constant vehicle speed of 50 km/h and an ambient temperature of 30°C.
Key Insights and Results
A critical factor in HVAC efficiency is the Degree of Subcool (DOS). Traditionally, finding the optimal DOS requires extensive manual tuning by human experts. The MADRL approach automates this by learning the optimal subcooling levels through its reward functions.
Performance Metrics
- Energy Efficiency: During the transient phase, the MADRL method consumed only 53% of the energy used by a tuned PID controller. In extreme cases, such as a 4°C target, the RL agent used only 11% of the energy compared to PID.
- Convergence Speed: The RL approach reached target temperatures significantly faster than PID, which often suffered from oscillations and slow settling times.
- Optimal Control: The system's self-taught steady-state behavior matched the efficiency of a "Grid Search"—a brute-force mathematical optimum—proving that the agents learned near-perfect control strategies.
Conclusion
This research demonstrates that MADRL can significantly extend EV range by optimizing HVAC energy consumption. While the current study focuses on Air Conditioning (A/C) mode, it provides a scalable foundation for more complex thermal management systems. Future work will involve validating these models in "Heat Pump" mode and testing on actual vehicle hardware to account for real-world sensor noise and thermal lag.
Summary updated