NFQ2.0: The CartPole Benchmark Revisited

Sascha Lange, Roland Hafner, Martin A. RiedmillerarXiv 2025 · DOI · arXiv

Abstract

Modernizing the 20-year-old Neural Fitted Q-iteration (NFQ) algorithm, this paper introduces NFQ2.0. By integrating modern deep learning practices—such as Adam optimizers, ReLU activations, and larger network architectures—NFQ2.0 achieves robust and repeatable performance on real-world industrial hardware. The study demonstrates that engineering choices like state stacking and action encoding are often more critical for industrial control success than algorithmic complexity.

Introduction

Deep Reinforcement Learning (Deep RL) is often perceived as "fickle"—highly sensitive to hyperparameters and difficult to transition from simulation to physical hardware. The original NFQ algorithm (2005) pioneered neural networks for batch RL but suffered from instability and required extensive manual tuning. This research revisits NFQ to bridge the gap between foundational RL theory and the practical requirements of industrial automation. The authors investigate whether the success of modern algorithms like DQN stems from their core mathematical updates or the "engineering" best practices surrounding them.

Methodology: Modernizing NFQ

NFQ2.0 retains the core "fitted" Q-learning loop while replacing 2005-era components with modern deep learning standards. Key upgrades include:

  • Network Architecture: Transitioning from small 20x20 networks to 256x256x100 Multi-Layer Perceptrons (MLP).
  • Optimizers & Activations: Replacing RProp with Adam and utilizing ReLU activations for better gradient flow.
  • Weight Persistence: Instead of re-training from scratch each iteration, NFQ2.0 continues training the same network, enhancing stability.
  • State Stacking: To mitigate the ~100ms latency in industrial fieldbus systems (which violates the Markov property), the authors stack the last six states and actions.
  • Action Encoding: Actions are encoded in the input layer, allowing the network to generalize to unseen actions between discrete training values.
Comparison of features between the original NFQ (2005), the modernized NFQ2.0, and DQN.

Experimental Setup

The performance of NFQ2.0 was evaluated across three environments:

  1. Simulated CartPole: A modified Gymnasium environment supporting the "swing-up" task.
  2. Physical CartPole: A custom real-world system built with standard industrial components, including a Siemens S1200 PLC and a V90 motor controller communicating via ProfiNet.
  3. Offline Learning: A demonstration of learning from a fixed dataset of 134 episodes to evaluate offline "reprogramming" capabilities.

Key Results and Insights

The "engineering over algorithm" hypothesis was confirmed: the core Bellman update remains effective when supported by modern optimizers (Adam), initialization (Glorot), and larger networks.

Performance and Stability

NFQ2.0 consistently learns successful swing-up and balance policies within 50 to 120 episodes, a significant improvement over the 392 episodes reported for the original NFQ. The resulting policies are highly precise, achieving balancing stability with a mean absolute deviation of just 1.25 to 1.69 degrees.

Learning curve showing average cost per step across 500 episodes, demonstrating rapid convergence and stability.

Engineering for the Real World

A critical finding was that state stacking is mandatory for physical hardware. Due to unavoidable system latency, a stack size of at least $n=2$ was required to learn a stable policy. Furthermore, the action-in-input architecture enabled Zero-Shot Action Generalization, where a policy trained on only three actions (Left, Neutral, Right) could immediately utilize 33 actions without additional training.

Conclusion and Future Work

NFQ2.0 demonstrates that "modernized vanilla" RL can be remarkably robust for industrial control. While the "windmill" phase (a local optimum where the pole spins rapidly) remains a challenge, the results suggest that foundational algorithms are often sufficient when paired with modern deep learning practices. Future work will explore whether these approaches scale to higher-dimensional state spaces and continuous action variants like NFQ-CA.

Summary updated

Abstract

This article revisits the 20-year-old neural fitted Q-iteration (NFQ) algorithm on its classical CartPole benchmark. NFQ was a pioneering approach towards modern Deep Reinforcement Learning (Deep RL) in applying multi-layer neural networks to reinforcement learning for real-world control problems. We explore the algorithm's conceptual simplicity and its transition from online to batch learning, which contributed to its stability. Despite its initial success, NFQ required extensive tuning and was not easily reproducible on real-world control problems. We propose a modernized variant NFQ2.0 and apply it to the CartPole task, concentrating on a real-world system build from standard industrial components, to investigate and improve the learning process's repeatability and robustness. Through ablation studies, we highlight key design decisions and hyperparameters that enhance performance and stability of NFQ2.0 over the original variant. Finally, we demonstrate how our findings can assist practitioners in reproducing and improving results and applying deep reinforcement learning more effectively in industrial contexts.
BibTeX
@misc{lange2025nfq,
  author = {Sascha Lange and Roland Hafner and Martin A. Riedmiller},
  title = {{NFQ2.0: The CartPole Benchmark Revisited}},
  year = {2025},
  eprint = {2511.12644},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.12644},
}
Figure