A reinforcement learning for continuous systems

Junichiro Yoshimoto, Shin Ishii, Masa-aki SatoNAIST 2000

Abstract

In this article, we propose a new reinforcement learning (RL) method for a system having continuous state and action spaces. Our RL method has an architecture like the actor-critic model. The critic tries to approximate the Q-function, which is the expected future return for the current state-action pair. The actor tries to approximate a stochastic soft-max policy defined by the Q-function. The soft-max policy is more likely to select an action that has a higher Q-function value. The on-line EM algorithm is used to train the critic and the actor. We apply this method to two control problems. Computer simulations show that our method is able to acquire fairly good control in the two tasks after a few learning trials.

Introduction

Reinforcement learning (RL) has demonstrated significant potential in discrete environments, such as board games or grid worlds. However, many real-world applications, particularly in robotics and control systems, operate within continuous state and action spaces. Adapting RL to these domains presents two primary challenges: the need for effective function approximation to represent infinite states and the difficulty of optimizing actions without computationally expensive or unstable gradient calculations.

This paper introduces a novel RL framework designed specifically for continuous domains. By moving away from standard gradient-descent updates toward a statistically robust on-line Expectation-Maximization (EM) learning scheme, we aim to achieve faster and more stable convergence in complex control tasks.

Methodology

The proposed architecture is an actor-critic model utilizing the Normalized Gaussian Network (NGNet) as a stochastic function approximator. The NGNet maps input vectors to outputs through a mixture of Gaussian basis functions, allowing for flexible representation of non-linear dynamics.

The Critic and the Q-Function

The critic's role is to approximate the Q-function, $Q(s, a)$, representing the expected future return for a given state $s$ and action $a$. Unlike traditional methods that rely on temporal difference (TD) gradients, our critic employs an on-line EM algorithm. This approach treats the selection of Gaussian units as hidden variables, providing more stable parameter updates by using discounted means to learn incrementally from environmental interactions.

The Actor and Soft-Max Policy

The actor is trained to approximate a stochastic soft-max policy derived from the critic’s Q-function. Instead of greedily pursuing the maximum Q-value, the actor matches a probability distribution where higher-reward actions are favored but exploration is maintained. This probabilistic matching helps prevent the agent from converging prematurely to local optima.

Experimental Evaluation

The effectiveness of the EM-based RL method was evaluated on two classic physical benchmarks: the inverted pendulum and the acrobot.

Inverted Pendulum

The goal is to swing a pole from a hanging position to a vertical upright position and balance it there, despite limited torque.

The Acrobot

The acrobot is a two-link, underactuated robot mimicking a gymnast on a high bar. Torque is applied only at the "elbow" joint (the connection between the two links). The objective is to swing the lower link above the pivot point and maintain balance.

Figure 2: Schematic of the Acrobot system showing links L1, L2 and angles q1, q2.

Results and Discussion

Simulation results indicate that the on-line EM algorithm significantly accelerates the learning process compared to traditional gradient-based methods.

Figure 1: Time sequence of the inverted pendulum control task, showing the swing-up and balancing phases.

In the inverted pendulum task, the agent achieved stable control in approximately 17 learning episodes. For the more complex acrobot task, successful balancing was achieved in 18 episodes. In comparison to previous gradient-based implementations—which required 40 and 37 episodes respectively—the proposed method effectively doubled the learning speed.

Conclusion

This research demonstrates that combining an actor-critic architecture with an on-line EM algorithm provides a powerful tool for RL in continuous spaces. The method offers superior convergence speed and stability over gradient-descent approaches. Future work will focus on addressing the computational overhead of EM updates and exploring the scalability of this approach to higher-dimensional systems and pixel-based inputs.

Summary updated

Abstract

In this article, we propose a new reinforcement learning (RL) method for a system having continuous state and action spaces. Our RL method has an architecture like the actor-critic model. The critic tries to approximate the Q-function, which is the expected future return for the current state-action pair. The actor tries to approximate a stochastic soft-max policy dened by theQ-function. The soft-max policy is more likely to select an action that has a higher Q-function value. The on-line EM algorithm is used to train the critic and the actor. We apply this method to two control problems. Computer simulations show that our method is able to acquire fairly good control in the two tasks after a few learning trials.
BibTeX
@inproceedings{yoshimoto2000reinforcement,
  author = {Junichiro Yoshimoto and Shin Ishii and Masa-aki Sato},
  title = {{A reinforcement learning for continuous systems}},
  booktitle = {NAIST},
  year = {2000},
}
Figure