Hierarchical Learning of Reactive Behaviors in an Autonomous Mobile Robot

Olac Fuentes, Rajesh P. N. Rao, Michael Van Wie

Abstract

This paper describes an autonomous mobile robot that employs a simple sensorimotor learning algorithm at three different behavioral levels to achieve coherent goal-directed behavior. The robot autonomously navigates to a goal destination within an obstacle-ridden environment by using the learned behaviors of obstacle-detection, obstacle-avoidance, and beacon-following. These reactive behaviors are learned in a hierarchical manner by using a hillclimbing routine that attempts to find the optimal transfer function from perceptions to actions for each behavior. Experimental results show that each behavior was successfully learned by the robot within a reasonably short period of time.

Introduction: The Challenge of Real-World Robotics

In the late 1990s, robotics researchers faced the "curse of dimensionality." As robots were equipped with more sensors and effectors, the space of possible sensorimotor mappings grew exponentially. Traditional solutions relied on hand-coding behaviors—which is brittle—or training in simulations. However, simulations often fail to capture the messy physics of the real world, such as friction or sensor noise, leading to policies that fail when deployed on actual hardware.

This research addresses how a robot can autonomously learn a transfer function from perceptions to actions in a high-dimensional space without relying on a simulator or exhaustive manual programming.

Hierarchical Architecture and Learning Algorithm

The researchers implemented a three-level hierarchical architecture inspired by Rodney Brooks’ subsumption architecture. Instead of attempting to learn a complex navigation task all at once, the robot learns behaviors in stages, with each layer building upon the stable performance of the layer below it.

The three-level hierarchical architecture for robot control, mapping specific sensors to reactive behaviors.

The Three Levels of Behavior

  1. Obstacle Detection: Uses five digital bump sensors to learn basic "get away" responses upon physical contact.
  2. Obstacle Avoidance: Uses three photosensors to detect darker areas (obstacles) and steer clear before a collision occurs.
  3. Beacon Following: Uses four infrared detectors to navigate toward a goal destination.

Heuristic Hillclimbing

The learning mechanism is a memory-efficient hillclimbing routine. For every perception-action pair, the robot maintains a heuristic value ($h$) representing the average "goodness" of that action. The robot receives a reward ($v$) based on its sensors (e.g., $v=1$ if no bumpers are pressed). It updates $h$ using a running average. If $h$ falls below a threshold, the robot assumes the current action is suboptimal and replaces it with a random new action, reinitializing the learning process for that specific state.

Experimental Setup

The "data" for this study was generated in real-time through physical interaction within a 1.2m x 1.2m robot arena. This enclosure contained various obstacles and an IR beacon. By learning directly on the hardware, the robot naturally accounted for physical realities, such as sensor blind spots, that a programmer might overlook in a simulation.

Key Results and Analysis

The authors provided quantitative evidence of learning across all three levels. The most fundamental metric was the collision rate during the initial learning phases.

Learning curve showing the sharp reduction in collisions per cycle as the robot masters obstacle detection.

As shown in the data, collisions per cycle dropped sharply and stabilized within approximately 500 cycles. Once the second layer (Obstacle Avoidance) was activated, the collision rate dropped even further as the robot learned to use photosensors to "see" obstacles before hitting them.

For the final task, beacon following, the robot showed dramatic improvement in efficiency. Initial trips to the beacon took over 110 seconds; however, after 35 trials, the robot optimized its pathing to reach the goal in just 40 seconds.

Conclusion and Future Work

The study demonstrates that hierarchical partitioning keeps the search space for each level manageable. While collisions were never reduced to zero—partly due to the robot's physical turning radius and the high density of obstacles—the robot successfully mastered goal-directed navigation through real-world interaction.

Future research directions include developing more sophisticated methods for autonomous coordination between behaviors and addressing how this approach might scale to continuous sensor data and more complex, non-reactive tasks.

Summary updated

Abstract

Describes an autonomous mobile robot that employs a simple sensorimotor learning algorithm at three different behavioral levels to achieve coherent goal-directed behavior. The robot autonomously navigates to a goal destination within an obstacle-ridden environment by using the learned behaviors of obstacle-detection, obstacle-avoidance, and beacon-following. These reactive behaviors are learned in a hierarchical manner by using a hillclimbing routine that attempts to find the optimal transfer function from perceptions to actions for each behavior. The authors present experimental results that show that each behavior was successfully learned by the robot within a reasonably short period of time. The authors conclude by discussing salient features of their approach and possible directions for future research.
BibTeX
@inproceedings{fuentes1997hierarchical,
  author = {Olac Fuentes and Rajesh P. N. Rao and Michael Van Wie},
  title = {{Hierarchical Learning of Reactive Behaviors in an Autonomous Mobile Robot}},
  booktitle = {Redalyc},
  year = {1997},
}
Figure