Abstract
Active thermal control is a critical challenge in engineering, often complicated by the nonlinear dynamics and time delays inherent in fluid flow. This paper investigates the use of Deep Reinforcement Learning (DRL) to regulate the heat transfer of a pulsating impinging jet. By integrating a Computational Fluid Dynamics (CFD) solver with various Deep Q-Network (DQN) architectures, the study demonstrates that DRL agents can effectively learn optimal cooling strategies. Specifically, "Soft" Double DQN and Dueling DQN variants maintained target temperatures for over 98% of the control cycle, significantly outperforming standard control approaches and vanilla DQN implementations.
Introduction
Thermal management is essential across diverse sectors, including aerospace, electric vehicle battery systems, and data center cooling. While passive cooling solutions are common, active cooling—where coolant flow is dynamically adjusted—offers superior efficiency. However, mastering active cooling is difficult because heat transfer is governed by complex Partial Differential Equations (PDEs) and involves significant thermal inertia. A change in coolant velocity does not result in an instantaneous temperature drop, creating a control challenge that traditional algorithms often struggle to solve.
This research explores whether DRL can navigate these complexities by interacting directly with a simulated environment to maintain a constant temperature on a heated surface.
Methodology
The researchers developed a 3D simulation environment using ANSYS FLUENT to model a hot plate cooled by an impinging jet. This CFD solver was coupled with a DRL agent in a closed-loop system to facilitate real-time learning.
The study evaluated several DQN architectures:
- Vanilla DQN: The baseline reinforcement learning model.
- Double DQN (Hard vs. Soft): Aimed at reducing overestimation bias. "Hard" updates involve periodic parameter copying, while "Soft" updates use a moving average for smoother transitions.
- Dueling DQN: Separately estimates state values and action advantages to improve learning efficiency.
The agent's objective was to adjust the jet velocity (action) to keep the plate's average surface temperature at a target of 303 K. The reward function was inversely proportional to the deviation from this target. The simulation utilized a structured mesh of $8 \times 10^5$ elements in a laminar flow regime ($Re = 170$ to $1700$).
Key Insights and Results
The experiments yielded several critical findings regarding the implementation of AI in fluid systems:
- Feedback Sensitivity: Sensor placement was paramount. Sensitivity analysis revealed that probes located very close to the wall ($1\text{ mm}$) provided the most effective feedback, whereas more distant sensors led to degraded agent performance due to increased signal delay.
- Stability of Soft Updates: The "Hard" Double DQN failed the task, producing high-frequency velocity oscillations. In contrast, the Soft Double DQN provided the necessary stability to handle the inertia-heavy nature of fluid flow, allowing for smooth control transitions.
- Performance Metrics: Both Soft Double DQN and Dueling DQN maintained the target temperature within a 2% deviation for more than 98% of the control cycle. These agents also improved temperature uniformity across the plate, effectively mitigating "hot spots" at the corners.
Conclusion
This study demonstrates that DRL is a powerful tool for complex thermal control tasks. By successfully managing the nonlinearities of impinging jets, the DRL agents proved more reliable and precise than traditional methods. Future research will focus on reducing computational costs through Reduced Order Models (ROMs) and extending these techniques to more chaotic turbulent flow regimes.
Summary updated