Deep Reinforcement Learning for the Heat Transfer Control of Pulsating Impinging Jets

Sajad Salavatidezfouli, Giovanni Stabile, Gianluigi RozzaarXiv 2023 · DOI · arXiv

Abstract

Active thermal control is a critical challenge in engineering, often complicated by the nonlinear dynamics and time delays inherent in fluid flow. This paper investigates the use of Deep Reinforcement Learning (DRL) to regulate the heat transfer of a pulsating impinging jet. By integrating a Computational Fluid Dynamics (CFD) solver with various Deep Q-Network (DQN) architectures, the study demonstrates that DRL agents can effectively learn optimal cooling strategies. Specifically, "Soft" Double DQN and Dueling DQN variants maintained target temperatures for over 98% of the control cycle, significantly outperforming standard control approaches and vanilla DQN implementations.

Introduction

Thermal management is essential across diverse sectors, including aerospace, electric vehicle battery systems, and data center cooling. While passive cooling solutions are common, active cooling—where coolant flow is dynamically adjusted—offers superior efficiency. However, mastering active cooling is difficult because heat transfer is governed by complex Partial Differential Equations (PDEs) and involves significant thermal inertia. A change in coolant velocity does not result in an instantaneous temperature drop, creating a control challenge that traditional algorithms often struggle to solve.

Schematic of the physical regions in an impinging jet flow, including the free jet, deflection, and wall jet regions.

This research explores whether DRL can navigate these complexities by interacting directly with a simulated environment to maintain a constant temperature on a heated surface.

Methodology

The researchers developed a 3D simulation environment using ANSYS FLUENT to model a hot plate cooled by an impinging jet. This CFD solver was coupled with a DRL agent in a closed-loop system to facilitate real-time learning.

The DRL-CFD framework showing the interaction between the DRL agent (DNN, Action Selection, Experience Replay) and the CFD environment.

The study evaluated several DQN architectures:

  • Vanilla DQN: The baseline reinforcement learning model.
  • Double DQN (Hard vs. Soft): Aimed at reducing overestimation bias. "Hard" updates involve periodic parameter copying, while "Soft" updates use a moving average for smoother transitions.
  • Dueling DQN: Separately estimates state values and action advantages to improve learning efficiency.

The agent's objective was to adjust the jet velocity (action) to keep the plate's average surface temperature at a target of 303 K. The reward function was inversely proportional to the deviation from this target. The simulation utilized a structured mesh of $8 \times 10^5$ elements in a laminar flow regime ($Re = 170$ to $1700$).

Key Insights and Results

The experiments yielded several critical findings regarding the implementation of AI in fluid systems:

  1. Feedback Sensitivity: Sensor placement was paramount. Sensitivity analysis revealed that probes located very close to the wall ($1\text{ mm}$) provided the most effective feedback, whereas more distant sensors led to degraded agent performance due to increased signal delay.
  2. Stability of Soft Updates: The "Hard" Double DQN failed the task, producing high-frequency velocity oscillations. In contrast, the Soft Double DQN provided the necessary stability to handle the inertia-heavy nature of fluid flow, allowing for smooth control transitions.
  3. Performance Metrics: Both Soft Double DQN and Dueling DQN maintained the target temperature within a 2% deviation for more than 98% of the control cycle. These agents also improved temperature uniformity across the plate, effectively mitigating "hot spots" at the corners.

Conclusion

This study demonstrates that DRL is a powerful tool for complex thermal control tasks. By successfully managing the nonlinearities of impinging jets, the DRL agents proved more reliable and precise than traditional methods. Future research will focus on reducing computational costs through Reduced Order Models (ROMs) and extending these techniques to more chaotic turbulent flow regimes.

Summary updated

Abstract

This research study explores the applicability of Deep Reinforcement Learning (DRL) for thermal control based on Computational Fluid Dynamics. To accomplish that, the forced convection on a hot plate prone to a pulsating cooling jet with variable velocity has been investigated. We begin with evaluating the efficiency and viability of a vanilla Deep Q-Network (DQN) method for thermal control. Subsequently, a comprehensive comparison between different variants of DRL is conducted. Soft Double and Duel DQN achieved better thermal control performance among all the variants due to their efficient learning and action prioritization capabilities. Results demonstrate that the soft Double DQN outperforms the hard Double DQN. Moreover, soft Double and Duel can maintain the temperature in the desired threshold for more than 98% of the control cycle. These findings demonstrate the promising potential of DRL in effectively addressing thermal control systems.
BibTeX
@misc{salavatidezfouli2023deep,
  author = {Sajad Salavatidezfouli and Giovanni Stabile and Gianluigi Rozza},
  title = {{Deep Reinforcement Learning for the Heat Transfer Control of Pulsating Impinging Jets}},
  year = {2023},
  eprint = {2309.13955},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.13955},
}
Figure