Incorporating Advice into Agents that Learn from Reinforcements

Richard Maclin, Jude W. ShavlikAAAI 1994

Abstract

Reinforcement learning (RL) is a powerful paradigm for autonomous agents, yet it is notoriously slow, often requiring thousands of trials to learn basic tasks from a "tabula rasa" (blank slate) state. This paper introduces a framework to "jumpstart" agents by allowing human observers to provide symbolic advice. This advice is compiled into a neural network using an extension of the Knowledge-Based Artificial Neural Networks (KBANN) algorithm, enabling the agent to benefit from human expertise while retaining the ability to refine or ignore that advice based on subsequent experience.

Introduction

The primary weakness of standard reinforcement learning is the exploration phase: in complex environments, an agent may wander aimlessly before encountering a reward. In contrast, human learning is a hybrid process involving both direct experience and instruction. The authors address the challenge of advice-taking: how to provide a connectionist Q-learner with high-level, symbolic instructions (e.g., "avoid box canyons") and have it effectively integrate that information into its numerical weights and biases.

The Advice-Taking Framework

The proposed system bridges the gap between human language and neural networks through a five-step process that allows an observer to intervene during the learning process.

The interaction between the learner, the environment, and the human observer providing advice.
  1. Request/Provide Advice: An observer monitors the agent's behavior and provides instructions when performance is suboptimal.
  2. Internal Representation: Advice is formulated in a simple, imperative programming language using domain-specific terms.
  3. Operationalization: A compiler translates "fuzzy" terms (e.g., "near") into concrete mathematical functions.
  4. Neural Integration: Using an incremental version of the KBANN algorithm, the compiled advice is inserted into the agent's existing Q-network as new hidden units. This allows advice to be added at any point during training.
  5. Refinement: The agent returns to standard RL, using backpropagation to fine-tune the newly installed units based on actual environmental rewards.

Symbolic Logic to Connectionist Weights

A key innovation of this work is the ability to map complex algorithmic structures, such as loops and multi-step plans, into a neural architecture. The system uses recurrent "state units" to maintain memory of previous actions, allowing the network to execute sequences defined in the advice language.

Examples of symbolic advice and their corresponding grid-world scenarios.

As shown in the figure above, advice can range from simple conditional actions (IF-THEN) to complex repetitive behaviors (REPEAT-UNTIL). These instructions do not act as hard constraints; instead, they serve as a "prior," biasing the agent toward certain actions while allowing the reinforcement learning process to eventually "wash out" bad advice if it leads to penalties.

Experimental Evaluation

The approach was tested in a 7x7 grid-world "video game" environment containing obstacles, rewards, and mobile enemies. The agent's task was to collect rewards and eliminate enemies by pushing obstacles into them. The researchers evaluated four distinct sets of advice:

  • SimpleMoves: Basic directional guidance.
  • NonLocalMoves: Guidance for navigating toward distant rewards.
  • ElimEnemies: Specific tactics for capturing enemies.
  • Surrounded: Escape maneuvers for when the agent is trapped.

Key Results and Insights

The experiments demonstrated that incorporating advice leads to statistically significant gains in performance across all metrics. For instance, the ElimEnemies advice tripled the mean number of enemies captured (from 0.15 to 0.44), while the Surrounded advice increased average survival time from 32.7 to 46.2 steps.

Crucially, the timing of the advice proved flexible; agents benefited whether the instruction was given at the start of training or after they had already gained significant experience. Furthermore, the system showed robustness: if the suggested advice was suboptimal, the reinforcement learning process successfully adjusted the network weights to prioritize more rewarding behaviors discovered through trial and error.

Conclusion

This research demonstrates a viable path for combining the strengths of symbolic AI and connectionist learning. By treating human advice as a refinable starting point rather than an immutable rule, agents can achieve high performance levels much faster than through pure reinforcement learning alone. Future work aims to explore the cumulative effects of multiple sequential pieces of advice and the potential for agents to evaluate advice against internal world models before execution.

Summary updated

BibTeX
@inproceedings{maclin1994incorporating,
  author = {Richard Maclin and Jude W. Shavlik},
  title = {{Incorporating Advice into Agents that Learn from Reinforcements}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1994},
}
Figure