Abstract
In this article, we propose a new reinforcement learning (RL) method for a system having continuous state and action spaces. Our RL method has an architecture like the actor-critic model. The critic tries to approximate the Q-function, which is the expected future return for the current state-action pair. The actor tries to approximate a stochastic soft-max policy defined by the Q-function. The soft-max policy is more likely to select an action that has a higher Q-function value. The on-line EM algorithm is used to train the critic and the actor. We apply this method to two control problems. Computer simulations show that our method is able to acquire fairly good control in the two tasks after a few learning trials.
Introduction
Reinforcement learning (RL) has demonstrated significant potential in discrete environments, such as board games or grid worlds. However, many real-world applications, particularly in robotics and control systems, operate within continuous state and action spaces. Adapting RL to these domains presents two primary challenges: the need for effective function approximation to represent infinite states and the difficulty of optimizing actions without computationally expensive or unstable gradient calculations.
This paper introduces a novel RL framework designed specifically for continuous domains. By moving away from standard gradient-descent updates toward a statistically robust on-line Expectation-Maximization (EM) learning scheme, we aim to achieve faster and more stable convergence in complex control tasks.
Methodology
The proposed architecture is an actor-critic model utilizing the Normalized Gaussian Network (NGNet) as a stochastic function approximator. The NGNet maps input vectors to outputs through a mixture of Gaussian basis functions, allowing for flexible representation of non-linear dynamics.
The Critic and the Q-Function
The critic's role is to approximate the Q-function, $Q(s, a)$, representing the expected future return for a given state $s$ and action $a$. Unlike traditional methods that rely on temporal difference (TD) gradients, our critic employs an on-line EM algorithm. This approach treats the selection of Gaussian units as hidden variables, providing more stable parameter updates by using discounted means to learn incrementally from environmental interactions.
The Actor and Soft-Max Policy
The actor is trained to approximate a stochastic soft-max policy derived from the critic’s Q-function. Instead of greedily pursuing the maximum Q-value, the actor matches a probability distribution where higher-reward actions are favored but exploration is maintained. This probabilistic matching helps prevent the agent from converging prematurely to local optima.
Experimental Evaluation
The effectiveness of the EM-based RL method was evaluated on two classic physical benchmarks: the inverted pendulum and the acrobot.
Inverted Pendulum
The goal is to swing a pole from a hanging position to a vertical upright position and balance it there, despite limited torque.
The Acrobot
The acrobot is a two-link, underactuated robot mimicking a gymnast on a high bar. Torque is applied only at the "elbow" joint (the connection between the two links). The objective is to swing the lower link above the pivot point and maintain balance.
Results and Discussion
Simulation results indicate that the on-line EM algorithm significantly accelerates the learning process compared to traditional gradient-based methods.
In the inverted pendulum task, the agent achieved stable control in approximately 17 learning episodes. For the more complex acrobot task, successful balancing was achieved in 18 episodes. In comparison to previous gradient-based implementations—which required 40 and 37 episodes respectively—the proposed method effectively doubled the learning speed.
Conclusion
This research demonstrates that combining an actor-critic architecture with an on-line EM algorithm provides a powerful tool for RL in continuous spaces. The method offers superior convergence speed and stability over gradient-descent approaches. Future work will focus on addressing the computational overhead of EM updates and exploring the scalability of this approach to higher-dimensional systems and pixel-based inputs.
Summary updated