Abstract
In industrial automation, balancing high-precision tracking with energy efficiency remains a critical challenge. This paper explores a Multi-Objective Reinforcement Learning (MORL) framework applied to a twin-rotor industrial testbed. By utilizing a composite reward function that weights tracking error against power consumption, we demonstrate that energy penalties act as a natural regularizer, promoting smoother control. However, our findings also reveal that standard optimization algorithms like Adam can introduce performance "dead zones" at low energy-penalty weights, leading to non-Pareto optimal solutions. Real-world validation on the Quanser Aero 2 hardware confirms that simulation-trained agents can successfully transition to physical systems, provided the objective weighting is properly tuned.
Introduction
Industrial control systems have traditionally faced a fundamental trade-off: maximizing performance (tracking accuracy) often comes at the cost of increased energy consumption. While classical control methods rely on precise mathematical models that are difficult to derive for non-linear systems, Reinforcement Learning (RL) offers a model-free alternative. However, standard RL typically optimizes for a single objective. This study investigates Multi-Objective Reinforcement Learning (MORL) to simultaneously address mission success and energy conservation in a 1-Degree-of-Freedom (1-DOF) helicopter-like system.
Methodology
Experimental Setup
The research utilized the Quanser Aero 2, a twin-rotor aerospace testbed. The system was configured for 1-DOF pitch control. To bridge the gap between software and hardware, the researchers developed a custom interface integrating a Simulink-based simulation environment with Python-based RL libraries.
Reward Design
The core of the MORL approach is a composite reward function designed to balance the competing objectives of tracking precision and power usage:
$$R_t = -(1 - \alpha) \cdot |\Delta_t| - \alpha \cdot P_t$$
Where:
- $\Delta_t$ represents the error in the pitch angle (tracking performance).
- $P_t$ represents the electrical power consumption.
- $\alpha$ (Alpha) is a weighting factor ranging from 0 to 1.
By adjusting $\alpha$, the agent's priority can be shifted from pure accuracy ($\alpha=0$) to total energy minimization ($\alpha=1$).
Key Insights and Results
The Regularization Effect of Energy
One of the primary findings is that penalizing energy consumption serves as a natural regularizer for the control policy. Agents trained solely for performance ($\alpha=0$) often developed aggressive "bang-bang" control behaviors, characterized by rapid voltage switching to maintain the target pitch. Introducing even a moderate energy penalty ($\alpha > 0.25$) forced the agents to adopt smoother, more stable control trajectories that are more suitable for industrial hardware longevity.
The Adam Optimizer Artifact
A significant discovery was the presence of a performance "cliff" at low values of $\alpha$ (between 0.0 and 0.25). In this range, the agents produced non-Pareto optimal solutions—meaning they performed worse in both accuracy and energy efficiency compared to agents trained with higher penalties.
The authors hypothesize that this is an artifact of the Adam optimizer. Adam’s adaptive learning rate may bias the learning process toward oscillatory, high-energy behaviors when the energy penalty is insufficient to counteract the tracking error gradient. This suggests that the choice of optimizer is as critical as the reward function itself in multi-objective tasks.
Sim-to-Real Validation
The learned strategies were deployed directly from the Simulink simulation to the physical Quanser Aero 2 hardware. The real-world results mirrored the simulation: agents with $\alpha \leq 0.25$ exhibited significant overshoots and persistent oscillations, while those with higher $\alpha$ values demonstrated efficient, human-like control. The study successfully mapped a Pareto front, indicating that substantial power savings can be achieved with only marginal sacrifices in tracking precision.
Conclusion and Future Work
This research demonstrates the viability of MORL for energy-aware industrial control while highlighting the subtle influence of optimization algorithms on policy stability. Future work will focus on:
- Automated Weighting: Utilizing Multi-Objective Bayesian Optimization (MOBO) to automatically determine the optimal $\alpha$ values.
- Optimizer Analysis: Investigating whether Stochastic Gradient Descent (SGD) or other optimizers can mitigate the aggressive control artifacts observed with Adam.
- Robustness: Enhancing sim-to-real transition techniques to better account for real-world sensor noise and environmental disturbances.
Summary updated