Hybrid Behavior Co-evolution and Structure Learning in Behavior-based Systems

Amir Massoud Farahmand, Majid Nili Ahmadabadi, Caro Lucas, Babak Nadjar AraabiIEEE CEC 2006 · OpenReview

Abstract

Designing an intelligent situated agent is a difficult task as the designer must see the problem from the agent's standpoint considering all its sensors and actuators. We have devised a co-evolutionary/reinforcement learning hybrid method to automate the design of hierarchical behavior-based systems. In our approach, the design problem is decomposed into two separate parts: developing a repertoire of behaviors and organizing those behaviors in a structure. Mathematical formulation shows how to decompose the value of the structure to simpler components. These components can be estimated and used to find the optimal organization of behaviors during the agent's lifetime. Moreover, a novel co-evolutionary mechanism is suggested that evolves each type of behavior separately in their own genetic pool. Our method is applied to the decentralized multi-robot object lifting task which results in human-competitive performance.

Introduction

Designing behavior-based systems—where a robot’s intelligence emerges from the interaction of simple, modular behaviors—is notoriously difficult. Traditionally, human designers use a "bottom-up" trial-and-error approach to decide which behaviors a robot needs and how they should be prioritized within a hierarchy. While researchers have attempted to automate this using artificial evolution or reinforcement learning (RL), both approaches have limitations: evolutionary robotics can be slow and produce non-modular "black box" controllers, while RL often converges to local optima in vast parameter spaces. This paper proposes a hybrid framework that evolves the "what" (the behaviors) and learns the "when" (the hierarchical structure).

Methodology

The authors propose a Hybrid Co-evolutionary/Reinforcement Learning method based on a "Baldwinian" mixture where evolution happens over generations, but learning happens within an individual agent's "lifetime."

  1. Behavior Co-evolution: The system maintains separate "genetic pools" for different types of behaviors (e.g., "go forward," "stop at goal"), promoting specialization.
  2. Structure Learning: Agents are formed by selecting behaviors from each pool. These agents then use Zero Order Representation reinforcement learning to find optimal priority levels within a Purely Parallel Subsumption Architecture (PPSSA).
  3. Fitness Sharing: Performance during the learning phase is returned to the genetic pools as a fitness score for the individual behaviors.
  4. Mathematical Decomposition: The total value of a structure is decomposed into simpler components that can be estimated stochastically, making hierarchy learning computationally efficient.

Experimental Setup

The method was evaluated using a simulated multi-robot object lifting task. Three robots must cooperatively lift a bulky, unknown object to a specific height while minimizing tilt. There is no central controller and no communication; robots rely entirely on local sensors for height, velocity, and tilt angle.

Visualization of the decentralized multi-robot object lifting task where three robots must cooperatively lift an unknown object.

Results and Discussion

The hybrid method was compared against three benchmarks: evolution of behaviors with a fixed structure, learning of structure with fixed behaviors, and a purely hand-designed system.

Fitness comparison of the hybrid co-evolutionary/learning method against various benchmarks over 50 generations.

Key findings include:

  • Human-Competitive Performance: The hybrid method (evolving behaviors + learning structure) reached performance levels comparable to the hand-designed system.
  • Superiority over Fixed Behaviors: The hybrid system outperformed the "hand-designed behaviors + learned structure" case, suggesting co-evolution finds more efficient internal mappings than human designers.
  • Learning as a Buffer: In early generations, hierarchy learning allows agents to adapt their structure to compensate for mediocre behaviors, guiding the evolutionary process more effectively than pure evolution.

Conclusion and Future Work

While successful, the current method faces "noisy" fitness assignment, as every behavior in an agent receives the same score. Future work will investigate more granular credit assignment based on specific rewards received during an agent's life. Additionally, using learned structures from previous generations as a "prior bias" could accelerate learning in new agents. While the lifting task proved successful, further testing in other multi-agent domains is required to demonstrate universality.

Summary updated

Abstract

Designing an intelligent situated agent is a difficult task as the designer must see the problem from the agent's standpoint considering all its sensors and actuators. We have devised a co-evolutionary/reinforcement learning hybrid method to automate the design of hierarchical behavior-based systems. In our approach, the design problem is decomposed into two separate parts: developing a repertoire of behaviors and organizing those behaviors in a structure. Mathematical formulation shows how to decompose the value of the structure to simpler components. These components can be estimated and used to find the optimal organization of behaviors during the agent's lifetime. Moreover, a novel co-evolutionary mechanism is suggested that evolves each type of behavior separately in their own genetic pool. Our method is applied to the decentralized multi-robot object lifting task which results in human-competitive performance.
BibTeX
@inproceedings{farahmand2006hybrid,
  author = {Amir Massoud Farahmand and Majid Nili Ahmadabadi and Caro Lucas and Babak Nadjar Araabi},
  title = {{Hybrid Behavior Co-evolution and Structure Learning in Behavior-based Systems}},
  booktitle = {IEEE CEC},
  year = {2006},
  url = {https://openreview.net/forum?id=mldvc9pJsqi},
}
Figure