A Reinforcement Learning Variant for Control Scheduling

Aloke GuhaNeurIPS 1990

Overview

This paper introduces "handicapped learning," a reinforcement learning (RL) variant designed to steer systems through a sequence of specific targets (setpoints) rather than just maintaining stability. By using a clever nonlinear state encoding and a "handicapped" mode that allows for exploration without immediate punishment, the algorithm can learn complex control schedules in a fraction of the time required by traditional methods.

The Challenge of Control Scheduling

In the early 90s, reinforcement learning excelled at "balancing" problems—like the classic pole-cart—where the goal is to keep a system within a stable region. However, many real-world industrial tasks require control scheduling: moving a system through a specific sequence of setpoints at specific times, such as changing the temperature of a chemical reactor or a building throughout the day.

Traditional RL faced two major hurdles in these environments:

  1. The Curse of Dimensionality: Using high-resolution linear state encoding for multiple setpoints leads to a combinatorial explosion of states, making learning impossibly slow.
  2. The Failure Paradox: In standard RL, being far from a target is often treated as a "failure." But if a system is intentionally moving from Setpoint A to Setpoint B, it must be far from its target for a while. Standard algorithms would "punish" the agent for this necessary transition, preventing it from learning the optimal trajectory.

Proposed Methods

The author proposes a strategy based on the Associative Search Element (ASE) and Adaptive Heuristic Critic (AHC) architecture, enhanced by three novel features:

  • Nonlinear State Encoding: Instead of a massive grid of states, the system uses a relative encoding. It defines a high-resolution "tolerance window" around the current desired setpoint ($X_d$) and only two additional states for being "too high" or "too low" outside that window. This keeps the state space small regardless of the total range of operation.
  • Handicapped Learning Mode: When the system needs to move to a new setpoint, it enters a "handicapped" mode. In this mode, the agent is allowed to explore action sequences to reach the new target without receiving the standard "failure" signal (negative reinforcement). This allows the controller to discover the necessary "steering" law.
  • Multivalued Reinforcement & Simulated Annealing: Unlike binary (success/fail) systems, this approach uses variable reinforcement. If the system fails in the same state repeatedly, the negative reinforcement increases—a technique inspired by simulated annealing to push the agent out of local minima.

Key Insights

  • Relative vs. Absolute States: By encoding states relative to the target rather than the environment, the agent learns a general "control law" (e.g., "if too cold, turn on heat") that applies across many different setpoints.
  • The "Handicap" as a Feature: Treating the transition between setpoints as a special mode prevents the "critic" from being too pessimistic. It acknowledges that the system is "under construction" during transitions.
  • Temporal Constraints: The introduction of a "timeout" in handicapped mode ensures the agent doesn't wander aimlessly; it must reach the new setpoint within a reasonable timeframe based on the distance it needs to travel.

Experimental Results

The paper evaluates the algorithm using two simulated control environments: a Continuous Stirred Tank Reactor (CSTR) and an Environmental Controller (HVAC) that must follow a daily time-temperature schedule.

The most striking results came from the HVAC environmental controller experiment:

  • Efficiency: The agent learned a complete daily schedule consisting of six time-temperature pairs (requiring 288 individual control steps) in just 43 trials.
Learning curve comparing Handicapped vs. No Handicapped Learning.

As shown in the learning curve above, the handicapped learning approach successfully masters the schedule, whereas traditional learning without the handicap fails to make meaningful progress. For context, a binary output controller could theoretically take up to $10^{86}$ trials to learn such a sequence through brute force.

  • Generalization: Once trained, the network was tested in a different environment with different exterior temperature profiles (and thus different cooling rates).
Test run demonstrating the controller's ability to generalize to new exterior temperature profiles.

The controller successfully generalized its learned heating/cooling laws to reach all six setpoints in the proper sequence, even under conditions it hadn't seen during training.

Limitations and Future Work

While highly effective, the approach has limitations. The current nonlinear encoding assumes the system's behavior is uniform across the control space. For highly nonlinear systems, an adaptive nonlinear encoding might be necessary. Additionally, the "handicapped" exploration is currently somewhat blind; if the system had prior information about the desired trajectory, learning could be even faster. Finally, the paper suggests exploring other RL formulations, specifically mentioning Watkins' Q-learning as a potential alternative to the ASE/AHC framework used here.

Summary updated

BibTeX
@inproceedings{guha1990reinforcement,
  author = {Aloke Guha},
  title = {{A Reinforcement Learning Variant for Control Scheduling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
}
Figure