An Adaptive Architecture for Modular Q-Learning

IJCAI 1997

TL;DR

This paper introduces AMQL (Automatic Modular Q-Learning), a novel architecture that allows reinforcement learning agents to automatically discover and adapt their internal modules to new environments. This overcomes the limitations of fixed modular architectures, enabling agents to learn more efficiently and adapt to dynamic changes without human intervention.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents learn to make decisions by maximizing rewards. While Q-learning is a popular RL technique, it struggles with scalability as problem complexity increases, requiring significant computational resources and learning time. Modular Q-learning attempts to address this by decomposing the problem into smaller subproblems handled by specialized modules. However, traditional modular approaches require human designers to pre-define these modules, making them inflexible and unable to adapt to changing environments. This paper tackles the challenge of creating RL agents that can autonomously adapt their internal structure to suit evolving environments.

Methods

The core innovation is the AMQL architecture, which operates in three phases:

  • Element Selection Phase: Each agent has a fixed number of modules. These modules randomly select a subset of environmental elements (sensors) to focus on, effectively partitioning the state space. Each module then builds its own Q-table based on this subset.
  • Fitness Allocation Phase: When an agent receives a reward, the modules that contributed to the rewarded action are assigned a "fitness" score. This score reflects how well a module's Q-values align with the chosen action. The paper uses the "greatest mass" (GM) strategy for action selection, where the action with the highest sum of Q-values across modules is chosen.
  • Module Selection Phase: Periodically, modules are evaluated based on their fitness. Modules with fitness above a certain threshold are retained, while those below are discarded. Discarded modules then re-enter the element selection phase to find a new focus. This process is likened to a genetic algorithm, where modules "evolve" to become more suitable.

Datasets

The paper uses the pursuit game as its simulation environment for evaluation. This is a standard benchmark in multi-agent RL, where multiple hunter agents attempt to capture a prey agent. The authors describe a specific 10x10 grid-based version of this game with toroidal edges. They also introduce a dynamic version of the pursuit game where the reward condition for capturing prey changes mid-experiment to test adaptability.

Key Insights and Ideas

  • Autonomous Module Discovery: The most significant insight is that agents can learn to construct effective modular structures without explicit human design. This shifts the burden from manual engineering to an automated learning process.
  • Adaptability through Reconfiguration: AMQL's ability to discard and re-select modules based on fitness allows it to adapt to dynamic environments where the optimal strategy or relevant environmental features change.
  • Trade-off between Speed and Optimality: While AMQL might take slightly longer to converge initially compared to a perfectly hand-designed modular system, it achieves a more robust and adaptable solution. The paper highlights that the "quality" of a module is determined by its fitness, which is directly tied to reward acquisition.
  • Modular Focus on Relevant Inputs: By selecting subsets of environmental elements, modules can specialize in processing specific aspects of the environment, leading to more efficient learning for those specialized tasks.

Key Results

  • Improved Learning Speed: AMQL significantly outperforms monolithic Q-learning in terms of learning speed, achieving capture in the pursuit game much faster.
  • Adaptability to Dynamic Environments: In a scenario where the target prey changed, AMQL agents successfully adapted and continued to capture prey, whereas fixed-module agents struggled significantly.
  • Comparison with Fixed Modules: While a well-designed fixed-module architecture (based on prior work) showed faster initial convergence, AMQL demonstrated superior adaptability. AMQL agents also consistently succeeded in obtaining suitable module sets, even if the initial learning was slightly slower than the optimal fixed-module case.

Limitations and Future Work

  • Initial Convergence Speed: The paper acknowledges that AMQL's initial convergence can be slower than a perfectly hand-designed fixed-module architecture because it spends time discovering the optimal module set.
  • Module Relevancy: When environments change drastically, AMQL might abandon modules that learned strategies no longer relevant. The authors suggest it would be beneficial to retain such modules for potential future use.
  • Future Work: The authors plan to investigate keeping previously learned modules, even if they are not immediately relevant, for potential reuse. They also suggest exploring the use of function approximators (like neural networks) within modules, rather than just lookup tables, to potentially improve learning convergence and adaptability further.

Summary updated

BibTeX
@inproceedings{1997adaptive,
  title = {{An Adaptive Architecture for Modular Q-Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1997},
}
Figure