Explanation-Based Neural Network Learning for Robot Control

Tom M. Mitchell, Sebastian ThrunNeurIPS 1992 · OpenReview

TL;DR

EBNN (Explanation-Based Neural Network Learning) bridges the gap between data-hungry neural networks and knowledge-rich symbolic AI. By using previously learned "action models" to calculate the derivatives (slopes) of a target function, EBNN allows robots to learn complex control policies from significantly fewer training examples than standard reinforcement learning.

What Problem Does This Paper Address?

Artificial neural networks are notoriously "data-hungry." To learn a complex function—like a robot's control policy—they typically require thousands of examples to generalize correctly. This is because standard neural networks rely on inductive bias, essentially performing a sophisticated form of smooth interpolation between isolated data points.

In the real world, especially in robotics, data is expensive and time-consuming to collect. Humans don't learn to drive by crashing a thousand times; we use prior knowledge about how steering affects direction to learn quickly. The authors aim to give neural networks a similar capability: using "domain theory" (previously learned knowledge about the world) to guide the learning of new tasks, thereby reducing the reliance on massive datasets.

Methods: Integrating Induction and Analysis

The authors introduce EBNN, which integrates two types of learning:

  1. Inductive Learning: The standard approach where the network fits a function to observed $(input, output)$ pairs (e.g., state-action pairs and their resulting rewards).
  2. Analytical Learning: This is the "explanation-based" part. When the robot observes a sequence of actions leading to a reward, it uses its "domain theory"—a set of pre-trained neural networks that model the effects of actions—to "explain" the outcome.
Action models chaining states to predict rewards

The "Aha!" Moment in the Math: Because neural networks are differentiable, the learner can use the chain rule to calculate the derivative of the final reward with respect to the initial state features. This provides "shape knowledge" (slopes). Instead of just knowing that a specific state is "good," the robot learns why it is good and how the reward would change if the state were slightly different.

Handling Imperfect Knowledge (LOB): A major risk in explanation-based learning is "garbage in, garbage out." If the prior knowledge is wrong, the analytical slopes will be wrong. To fix this, the authors introduced the LOB heuristic. It weights the analytical component based on how accurately the domain theory predicted the actual observed state. If the prediction is poor, the system ignores the analytical "advice" and relies on pure induction.

Key Insights and Ideas

  • Slopes are Better than Points: A single training point tells you the value of a function at one spot. A training point plus a slope tells you the trend of the function in that neighborhood. This extra information is what allows EBNN to generalize so much faster.
Comparing function fitting with and without slope constraints
  • Differentiable Explanations: Traditional Explanation-Based Learning (EBL) used symbolic logic (if-then rules), which is brittle. By using neural networks as the "rules," EBNN makes the process continuous and differentiable, allowing it to handle the "fuzziness" of the real world.
  • Graceful Degradation: Thanks to the LOB* weighting, EBNN is robust. If the robot enters a part of the environment it doesn't understand yet, it doesn't get "confused" by its bad internal models; it simply reverts to standard trial-and-error learning until its models improve.

Datasets and Key Results

The method was evaluated in a simulated robot navigation domain where the robot had to navigate to a goal while avoiding obstacles using five discrete actions. To test robustness, the authors used nine different versions of action models, pre-trained on varying amounts of data (from 5 to 8,192 examples).

  • Efficiency: EBNN significantly outperformed pure inductive learning (standard Q-learning). It reached high success rates with far fewer episodes.
  • Knowledge Transfer: The experiments showed that more accurate domain theories (those trained on more data) led to faster learning of the control policy.
  • Robustness: Even when the domain theory was extremely weak (trained on only 5 examples), EBNN eventually matched the performance of pure induction. It didn't "break" the learner, proving the effectiveness of the LOB* weighting mechanism.

Limitations and Future Work

  • Discrete Actions: The current implementation assumes a discrete action space. The authors suggest extending this to real-valued, parameterized actions.
  • Stochasticity: The simulated environment was deterministic. Performance in highly stochastic environments remains an open question.
  • Complexity of Explanations: The paper focuses on first-order derivatives (slopes). Future work could explore whether higher-order derivatives (curvature) could provide even more useful constraints for learning.
  • Scaling: While successful in simulation, the authors mention the need for more research on automatically modularizing domain theories to handle much larger, more complex tasks.

Summary updated

Abstract

How can artificial neural nets generalize better from fewer examples? In order to generalize successfully, neural network learning methods typically require large training data sets. We introduce a neural network learning method that generalizes rationally from many fewer data points, relying instead on prior knowledge encoded in previously learned neural networks. For example, in robot control learning tasks reported here, previously learned networks that model the effects of robot actions are used to guide subsequent learning of robot control functions. For each observed training example of the target function (e.g. the robot control policy), the learner explains the observed example in terms of its prior knowledge, then analyzes this explanation to infer additional information about the shape, or slope, of the target function. This shape knowledge is used to bias generalization when learning the target function. Results are presented applying this approach to a simulated robot task based on reinforcement learning.
BibTeX
@inproceedings{mitchell1992explanation,
  author = {Tom M. Mitchell and Sebastian Thrun},
  title = {{Explanation-Based Neural Network Learning for Robot Control}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1992},
  url = {https://openreview.net/forum?id=S1-HsUZu-H},
}
Figure