Abstract
We explore the use of learning schemes in training and adapting performance on simple coordination tasks. The tasks are 1-D pole balancing. Several programs incorporating learning have already achieved this (1, S, 8): the problem is to move a cart along a short piece of track to at to keep a pole balanced on its end; the pole is hinged to the cart at its bottom, and the cart is moved either to the left or to the right by a force of constant magnitude. The form of the task considered here, after (3), involves a genuinely difficult credit-assignment problem. We use a learning scheme previously developed and analysed (1, 7) to achieve performance through reinforcement, and extend it to include changing and new requirements. For example, the length or mast of the pole can change, the bias of the force, its strength, and so on; and the system can be tasked to avoid certain regions altogether. In this way we explore the learning system's ability to adapt to changes and to profit from a selected training sequence, both of which are of obvious utility in practical robotics applications.
Introduction
In the mid-1980s, robotic control was dominated by traditional control theory, which often struggled with unforeseen changes in dynamics or task requirements. This research addresses the credit-assignment problem: in a task like pole-balancing, if the pole falls after a long sequence of successful steps, which specific movement was the error? When feedback is "sparse"—only receiving a signal upon total failure—learning becomes incredibly slow. This study investigates whether a system can "track" a moving optimum by adapting to physical changes and whether "directed training" (curriculum learning) can outperform starting from scratch.
Methodology and System Configuration
The researchers utilized a computer simulation of a 1-D Cart-Pole (Inverted Pendulum) system. The physical parameters defining the simulation environment, including mass and track length, are shown below:
To manage the control task, the continuous state space—comprising cart position ($x$), pole angle ($\theta$), and their respective velocities—was discretized into 162 distinct regions. The specific ranges for these variables are detailed below:
The learning algorithm assigned actions to these regions. Rather than relying solely on failure signals, it employed a "back-propagation" method to generate an internal evaluation signal. This allowed the system to identify and avoid regions close to failure before a crash occurred, effectively transforming a flat performance landscape (the "mesa phenomenon") into a navigable gradient.
Key Results
Adaptation vs. Re-learning
The study found that modifying an existing solution is significantly more efficient than starting over. For example, training a system to balance a 1kg pole from scratch required 86 failures. In contrast, a system previously trained on a 0.1kg pole (68 failures) adapted to the 1kg pole with only 20 additional failures.
Directed Training (Curriculum Learning)
Directed training proved essential for difficult constraints. Balancing on a restrictive 2-meter track resulted in over 250 failures when attempted from scratch. However, by following a graded sequence of track lengths (4.8m → 4m → 3m → 2m), the system achieved stability (balancing for 10,000 steps) in only 82 total failures.
Constraint Satisfaction
When penalized for remaining near the vertical center, the system successfully learned to balance at an angle, reducing time in the "forbidden" zone from 22% to 8% without explicit instructions on how to achieve this.
Conclusion
This work demonstrates that robots can discover how to achieve goals specified only in terms of what (e.g., "avoid falling"). This "learning by discovery" is robust to physical changes like mechanical wear. The authors suggest that future robotic systems should incorporate a hierarchy of purposes, such as refining movements for efficiency once basic stability is mastered.
Summary updated