Abstract
The Organic Rankine Cycle (ORC) is widely used in industrial waste heat recovery due to its simple structure and easy maintenance. However, in the context of smart manufacturing, traditional model-based optimization control methods struggle to adapt to varying operating conditions or sudden mode changes. Deep reinforcement learning (DRL) offers significant advantages in uncertain environments by interacting directly with the system without requiring an explicit model. Nevertheless, direct application of DRL to physical ORC systems presents unacceptable safety risks, and its generalization performance under model-plant mismatch is often insufficient. This paper proposes a Sim2Real transfer learning-based DRL control method for ORC superheat control. By utilizing a surrogate model for pre-training, the proposed method improves training speed and solves generalization issues under multiple operating conditions. Experimental results demonstrate that this approach provides a feasible and user-friendly solution for energy system optimization.
Introduction: The Challenge of ORC Control
Organic Rankine Cycle (ORC) systems are essential for converting low-grade industrial waste heat into electricity. Despite their utility, they are notoriously difficult to control because the waste heat source—characterized by fluctuating temperatures and flow rates—is highly unstable. Furthermore, the internal thermodynamics involve complex phase changes that are difficult to model accurately.
Traditional methods like Proportional-Integral-Derivative (PID) control are often too "passive," reacting only after an error occurs. Model Predictive Control (MPC) offers better performance but requires a high-fidelity mathematical model that is difficult to maintain. Deep Reinforcement Learning (DRL) provides a model-free alternative, but its "trial and error" nature poses two major hurdles for industrial application: safety risks to expensive hardware and the extreme time requirements for training on physical systems.
The ORC System Architecture
The controlled plant in this study is a standard ORC system designed for waste heat recovery. The cycle consists of four primary components: a pump, an evaporator, an expander, and a condenser.
As shown in the figure above, the working fluid is pressurized by the pump and then heated in the evaporator by an external heat source. The resulting high-pressure vapor drives the expander to produce power. Finally, the fluid is cooled in the condenser before returning to the reservoir. Maintaining the proper superheat (SH) at the evaporator outlet is critical for system efficiency and preventing liquid droplets from damaging the expander.
Proposed Sim2Real Framework
To overcome the limitations of direct DRL application, the authors developed a three-stage Surrogate Empowered Sim2Real framework:
1. Surrogate Modeling (The Virtual Prototype)
Instead of relying on complex physics-based equations, the researchers used 10,000 points of historical operational data to train a Long Short-Term Memory (LSTM) neural network. This LSTM acts as a "virtual prototype" (VP), accurately predicting the system's next state based on current conditions and control inputs. This surrogate model achieved a remarkably low Mean Squared Error (MSE) of 0.0265.
2. Offline Pre-training
A DRL agent was trained using the Proximal Policy Optimization (PPO) algorithm within this virtual environment. This allowed the agent to experience thousands of simulated episodes of fluctuating heat conditions in a fraction of the time required in reality, without any risk to physical hardware.
3. Online Sim2Real Transfer & Fine-tuning
The pre-trained agent was then transferred to the actual ORC system. To ensure safety during this phase, the authors implemented Engineered Exploration. By reducing the exploration variance, the agent was prevented from taking extreme actions while it fine-tuned its policy to account for the "model-plant mismatch" between the simulation and the real world.
Key Results and Insights
The Sim2Real approach yielded significant improvements over traditional and standard DRL methods:
- Training Efficiency: The Sim2Real agent achieved high rewards within just a few dozen episodes, whereas an agent learning from scratch required thousands of episodes to reach comparable performance.
- Superior Control Performance: In head-to-head tests against a well-tuned PI controller, the DRL agent demonstrated near-zero overshoot and significantly faster response times during setpoint changes and heat source fluctuations.
- Proactive Disturbance Rejection: Unlike PID controllers that react to errors, the DRL agent learned to anticipate disturbances in the waste heat flow, adjusting the pump speed proactively to maintain stable superheat levels.
Conclusion and Future Work
This research demonstrates that historical operational data can be leveraged to build high-fidelity virtual environments for safe DRL training. The proposed Sim2Real framework effectively bridges the gap between simulation and reality, providing a robust solution for industrial heat recovery control. Future work will focus on automating the tuning of exploration parameters and extending the framework to multi-objective optimization, such as maximizing net power output alongside safety constraints.
Summary updated