Abstract
Modernizing the 20-year-old Neural Fitted Q-iteration (NFQ) algorithm, this paper introduces NFQ2.0. By integrating modern deep learning practices—such as Adam optimizers, ReLU activations, and larger network architectures—NFQ2.0 achieves robust and repeatable performance on real-world industrial hardware. The study demonstrates that engineering choices like state stacking and action encoding are often more critical for industrial control success than algorithmic complexity.
Introduction
Deep Reinforcement Learning (Deep RL) is often perceived as "fickle"—highly sensitive to hyperparameters and difficult to transition from simulation to physical hardware. The original NFQ algorithm (2005) pioneered neural networks for batch RL but suffered from instability and required extensive manual tuning. This research revisits NFQ to bridge the gap between foundational RL theory and the practical requirements of industrial automation. The authors investigate whether the success of modern algorithms like DQN stems from their core mathematical updates or the "engineering" best practices surrounding them.
Methodology: Modernizing NFQ
NFQ2.0 retains the core "fitted" Q-learning loop while replacing 2005-era components with modern deep learning standards. Key upgrades include:
- Network Architecture: Transitioning from small 20x20 networks to 256x256x100 Multi-Layer Perceptrons (MLP).
- Optimizers & Activations: Replacing RProp with Adam and utilizing ReLU activations for better gradient flow.
- Weight Persistence: Instead of re-training from scratch each iteration, NFQ2.0 continues training the same network, enhancing stability.
- State Stacking: To mitigate the ~100ms latency in industrial fieldbus systems (which violates the Markov property), the authors stack the last six states and actions.
- Action Encoding: Actions are encoded in the input layer, allowing the network to generalize to unseen actions between discrete training values.
Experimental Setup
The performance of NFQ2.0 was evaluated across three environments:
- Simulated CartPole: A modified Gymnasium environment supporting the "swing-up" task.
- Physical CartPole: A custom real-world system built with standard industrial components, including a Siemens S1200 PLC and a V90 motor controller communicating via ProfiNet.
- Offline Learning: A demonstration of learning from a fixed dataset of 134 episodes to evaluate offline "reprogramming" capabilities.
Key Results and Insights
The "engineering over algorithm" hypothesis was confirmed: the core Bellman update remains effective when supported by modern optimizers (Adam), initialization (Glorot), and larger networks.
Performance and Stability
NFQ2.0 consistently learns successful swing-up and balance policies within 50 to 120 episodes, a significant improvement over the 392 episodes reported for the original NFQ. The resulting policies are highly precise, achieving balancing stability with a mean absolute deviation of just 1.25 to 1.69 degrees.
Engineering for the Real World
A critical finding was that state stacking is mandatory for physical hardware. Due to unavoidable system latency, a stack size of at least $n=2$ was required to learn a stable policy. Furthermore, the action-in-input architecture enabled Zero-Shot Action Generalization, where a policy trained on only three actions (Left, Neutral, Right) could immediately utilize 33 actions without additional training.
Conclusion and Future Work
NFQ2.0 demonstrates that "modernized vanilla" RL can be remarkably robust for industrial control. While the "windmill" phase (a local optimum where the pole spins rapidly) remains a challenge, the results suggest that foundational algorithms are often sufficient when paired with modern deep learning practices. Future work will explore whether these approaches scale to higher-dimensional state spaces and continuous action variants like NFQ-CA.
Summary updated