Reinforcement Learning for Mixed Open-loop and Closed-loop Control

Eric A. Hansen, Andrew G. Barto, Shlomo ZilbersteinNeurIPS 1996

Introduction

Standard reinforcement learning (RL) often operates under the assumption that "looking" at the environment is free. In traditional RL, agents operate in a continuous "closed-loop": they observe the state, take an action, and repeat. This assumes that sensory feedback is instantaneous and without penalty. However, in the real world, sensing costs time, energy, or money. For a robot deployed in the field or a complex industrial process, sensing can be expensive due to the energy required to run sensors, the computational cost of processing data (like computer vision), or the opportunity cost of pausing a task to perform a measurement.

If sensing carries a cost, the agent must decide not just what to do, but when to look. This creates a "mixed" control problem. Between observations, the agent must act "open-loop" (blindly), which naturally leads to increasing uncertainty about its true state. While this scenario can be framed as a Partially Observable Markov Decision Process (POMDP), POMDPs are notoriously difficult and computationally expensive to solve. This paper introduces a Q-learning variant designed to efficiently learn an optimal sensing interval, interleaving "blind" open-loop action sequences with "informed" closed-loop sensing.

Methods: Augmented State Spaces and Pruning

To tackle this computationally, the authors extend standard Q-learning by formalizing the problem as an MDP with an augmented state space. They introduce three core concepts:

  1. Action Separation: The framework distinguishes between control actions (which change the environment but provide no new information) and sensing actions (which provide perfect state information but do not alter the environment).
  2. Memory States: To handle the "blind" periods between sensing, the agent utilizes "memory states." A memory state is a sequence representing the last known state and every action taken since that observation (e.g., "I was at State X, and I have since moved North, then East").
Tree of memory states representing open-loop action sequences
  1. The Pruning Rule: The primary innovation of the paper is a heuristic rule designed to manage the combinatorial explosion of possible action sequences (illustrated by the expanding tree of memory states). The agent maintains two estimates:
  2. $g$: The expected reward for the current open-loop sequence.
  3. $h$: The discounted expected value of information (the value of sensing now).

If the current estimated value of the state is already better than what the agent expects to achieve by continuing the current open-loop branch, it prunes that branch. Essentially, if the "cost of curiosity" outweighs the "value of information," the agent stops exploring that specific sequence and triggers a sensing action. The authors bridge RL with heuristic search, showing that this pruning rule is analogous to $f = g + h$ in A* search.

Experimental Setup

Rather than using large-scale datasets, the algorithm is evaluated on a stochastic Grid World simulation designed to test the cost-benefit analysis of sensing:

  • The Task: A robot must navigate to a goal in a 20-state grid.
  • The Twist: Actions are stochastic. There is a 10% chance of the agent not moving, and a 10% chance of it moving sideways relative to its intended direction.
  • The Costs: Sensing costs 1 unit, moving costs 4 units, and reaching the goal yields a reward of 100 units.

Key Results and Insights

By treating "observation" as a deliberate action with an associated cost, the agent naturally learns to act blindly when the environment is predictable and "look" only when uncertainty becomes too risky.

The 20-state Grid World environment and the optimal mixed strategy learned by the agent
  • Optimal Strategy: The agent successfully learned a mixed strategy. As shown in the results, it takes multiple steps in "safe" areas of the grid before stopping to sense its location when near obstacles or the goal (indicated by the 'O' for observe in the action sequences).
  • Computational Efficiency: In the Grid World example, a brute-force approach with a sequence limit of five actions would require exploring over 64,000 memory states. Increasing the limit to nine actions pushes this to 16 million.
  • Pruning Power: By utilizing the proposed pruning rule, the agent converged to an optimal policy by exploring fewer than 1,000 states, regardless of how long the potential action sequences were allowed to be.
  • Convergence Guarantee: A significant theoretical contribution of the paper is a proof demonstrating that Q-learning still converges to an optimal policy even when massive portions of the state space are never explored due to this pruning.

Limitations and Future Work

The primary limitation of this model is its assumption of perfect information: it assumes that when the agent does choose to sense, the resulting information is 100% accurate. In many real-world robotics scenarios, sensors are noisy, which would necessitate a more complex POMDP approach. Furthermore, the proof and methods are designed for discrete states and actions. Extending this to continuous spaces would require function approximation techniques. Future directions suggested by the authors include applying similar heuristic pruning rules to more general POMDPs where sensors provide imperfect or incomplete information.

Summary updated

BibTeX
@inproceedings{hansen1996reinforcement,
  author = {Eric A. Hansen and Andrew G. Barto and Shlomo Zilberstein},
  title = {{Reinforcement Learning for Mixed Open-loop and Closed-loop Control}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1996},
}
Figure