Abstract
Reinforcement Learning (RL) offers a promising framework for controlling complex, non-linear Industrial Cyber-Physical Systems (ICPSs) without requiring explicit mathematical models. However, the transition from theoretical research to industrial deployment is often hindered by poor problem formulation. This paper investigates how structured environment design—specifically normalization, domain randomization, and reward shaping—impacts the training stability and sample efficiency of RL agents. Using a 1-DoF helicopter testbed, we demonstrate that a well-formulated Markov Decision Process (MDP) enables agents to be trained from scratch on physical hardware, overcoming common pitfalls such as "bang-bang" control and training instability.
Introduction
In the realm of Industrial Cyber-Physical Systems (ICPSs), Reinforcement Learning (RL) is frequently touted as a transformative technology. Its ability to learn control policies through interaction makes it ideal for systems where traditional control theory struggles with non-linear dynamics or modeling inaccuracies. Despite this potential, RL remains underutilized in actual factory settings.
The primary bottleneck is not necessarily the choice of algorithm, but rather the engineering workflow surrounding the problem formulation. While supervised learning benefits from standardized data preprocessing, RL often relies on ad-hoc definitions of the state space, action space, and reward functions. This paper argues that prioritizing the design of the Markov Decision Process (MDP) is essential for bridging the gap between simulation and real-world application.
Methodology and Environment Design
The study utilizes Proximal Policy Optimization (PPO), a robust model-free RL algorithm, to control a Quanser Aero 2 (a 1-DoF helicopter system). The core contribution lies in the transition from a "Baseline" formulation to a "New setting" based on five foundational design principles:
- Observation and Action Normalization: Scaling all inputs (pitch, velocity) and outputs (voltage) to a strict $[-1, 1]$ range. This ensures that the neural network's gradients remain stable and prevents specific features from dominating the learning process.
- Randomized Target Trajectories: Instead of training on a static goal, the target pitch angle is varied randomly. This prevents the agent from memorizing specific trajectories and forces the development of a generalized control law.
- Extended Episode Horizons: The simulation horizon was increased significantly (from 80 to 100,000 seconds). Longer episodes allow the agent to experience a more continuous and diverse range of the state space.
- Random Initial States: By starting the system at various pitch angles, the agent is forced to learn recovery and stabilization strategies across the entire operational envelope from the start of training.
- Action Penalties (Reward Shaping): To prevent hardware damage, a penalty was introduced for high-frequency voltage fluctuations. This discourages "bang-bang" control—a common RL artifact where the agent rapidly switches between maximum and minimum control signals.
Experimental Results
The effectiveness of these principles was evaluated through ablation studies in a Simulink environment and subsequent deployment on physical hardware.
Simulation Performance and Ablation
The "New setting" demonstrated remarkable sample efficiency, reaching the target performance threshold (average deviation $\le 4^\circ$) in only 40,000 steps. This was achieved with a 100% success rate across multiple random seeds. In contrast, the "Baseline" formulation failed to reach this threshold reliably.
Ablation studies revealed that normalization was the most critical factor. When normalization was removed from the optimized setting, the steps required to reach the performance goal increased from 40,000 to 595,000. This highlights how simple numerical scaling can be more impactful than complex hyperparameter tuning.
Real-World Deployment
A significant milestone of this research was the successful training of the PPO agent directly on the physical Quanser Aero 2 hardware. By employing the optimized problem formulation, the agent learned an effective control policy from scratch in 200,000 steps. Crucially, the inclusion of action penalties ensured that the learned voltage signals were smooth and safe for the physical motors, avoiding the aggressive control behaviors often seen in simulation-only agents.
Discussion and Insights
The results suggest that "simulation success" can be deceptive. Without proper reward shaping (like action penalties), an agent may find mathematically optimal solutions in a simulator that are physically destructive. Furthermore, the synergy between different formulation tweaks—such as combining randomization with normalization—creates a more robust learning landscape than any single modification alone.
Conclusion and Future Work
This work underscores that the success of RL in industrial applications depends heavily on the "engineering" of the environment rather than just the "tuning" of the algorithm. By adopting structured principles for MDP design, practitioners can achieve higher sample efficiency and safer real-world performance.
Future research will focus on scaling these principles to higher-degree-of-freedom systems and integrating more sophisticated control metrics, such as steady-state error and rise-time, into the RL reward structure. Additionally, exploring the intersection of physics-informed RL and Model Predictive Control (MPC) remains a promising avenue for creating even more reliable industrial controllers.
Summary updated