Abstract
Steel production via Electric Arc Furnaces (EAFs) is a critical decarbonization strategy but is hindered by high energy costs and volatile electricity markets. This paper introduces a reinforcement learning (RL) framework for EAF scheduling that utilizes a Q-learning approach with a novel "penalty smoothing" reward function. The proposed method achieves approximately 90% of the profit of a perfect-foresight optimization model while executing 500 times faster, offering a scalable solution for real-time industrial demand response.
Introduction
Steel production accounts for 7-9% of global greenhouse gas emissions. Transitioning from traditional blast furnaces to Electric Arc Furnaces (EAFs)—which use scrap steel and electricity—is a primary decarbonization strategy. However, EAFs are extremely energy-intensive. To be economically viable, they must employ "demand response" strategies, shifting heavy loads to periods when electricity is cheap.
The challenge lies in scheduling complexity, involving non-convex start-up costs, specific "dwell-time" requirements, and grid power constraints. Traditional Mixed-Integer Linear Programming (MILP) methods often assume "perfect foresight" of future prices and are computationally expensive for real-time applications. This research seeks a way to make smart, real-time decisions using available data, such as day-ahead price forecasts from the New York Independent System Operator (NYISO), without requiring excessive computational resources.
Methodology
The researchers developed a two-pronged approach to benchmark their solution. A rolling-horizon MILP model serves as the "gold standard," representing the maximum possible profit if an operator had perfect knowledge of future price spikes. The proposed solution is a tabular Q-learning agent that operates using only current furnace states and day-ahead price (DAP) signals.
Key innovations in the RL approach include:
- Reward Smoothing: Instead of applying a massive startup penalty at once, the cost is distributed across active melting and tapping steps to stabilize learning.
- Heuristic Tie-breaking: When multiple actions have the same predicted value, the agent uses a short-horizon template-based lookahead to choose the most logical sequence for a furnace cycle.
- Feasibility Masking: To ensure the agent never suggests an impossible action, the action space is filtered based on physical constraints before a choice is made.
Results and Discussion
The Q-learning agent demonstrated significant performance, capturing 89.8% of the profit of the perfect-foresight MILP model and providing a 72.4% improvement over a fixed-cycle baseline.
A critical advantage for industrial deployment is computational speed. While the MILP takes about 50ms per step, the RL inference takes only 0.09ms—roughly 500 times faster. This efficiency allows for instantaneous response to market shifts. The RL policy effectively "shaves the peaks" by pausing during price spikes and "fills the valleys" by maximizing throughput when energy is cheap.
Furthermore, the agent demonstrated emergent coordination. Even without a global optimization objective, it learned to stagger the melting phases of multiple furnaces, preventing the facility from exceeding its total power cap and avoiding peak demand penalties.
Conclusion
This RL framework provides a robust and computationally efficient alternative to traditional scheduling methods for EAFs. While the current study uses simplified physics and depends on forecast accuracy, the results indicate that RL can effectively manage complex industrial loads under price volatility. Future work will focus on scaling to larger facilities using Deep Reinforcement Learning (DRL) to handle higher-dimensional state spaces and integrating more complex thermochemical dynamics.
Summary updated