Symbol Grounding and Task Learning from Imperfect Corrections

Mattias Appelgren, Alex LascaridesACL 2021

TL;DR

This paper tackles the challenge of teaching robots new tasks through natural language, even when the human teacher makes mistakes. They introduce a novel graphical model that allows agents to learn symbol grounding (like object colors) and task goals, even when the teacher's feedback is unreliable, leading to more robust learning systems.

What Problem Does This Paper Address?

Interactive Task Learning (ITL) aims to equip agents with the ability to learn new tasks through interaction with a human teacher. While previous work often assumes a perfect teacher who provides timely and accurate feedback, humans are prone to errors. This paper addresses the critical gap of learning from imperfect corrective feedback. Specifically, it focuses on scenarios where a teacher might:

  • Fail to provide a correction when an agent makes a mistake.
  • Provide a correction for an action that was actually correct, or for a mistake made earlier than the most recent action.
  • Use ambiguous language that could refer to multiple rules or concepts.

These imperfections can lead to the agent misinterpreting the teacher's intent, misattributing corrections, and ultimately failing to learn the task or ground symbols (like object colors) correctly. The goal is to build agents that can learn effectively despite these human limitations.

Methods

The core of the paper's contribution lies in a probabilistic graphical model designed to handle imperfect teacher feedback. Key aspects include:

  • Discourse Coherence: The model leverages the principle that utterances in a dialogue should be coherent with the preceding context. In this ITL setting, a correction utterance is expected to relate to a previous agent action.
  • Separating Observable vs. Latent Events: A crucial innovation is distinguishing between the observable event of a teacher uttering a correction (TeacherCorrection(a, u)) and the latent event of that correction being truly coherent with the agent's action (CC(a, u)). This separation allows the agent to reason about the teacher's potential fallibility.
  • Probabilistic Reasoning: The model uses Bayesian inference to update beliefs about:
  • Symbol Grounding: Learning the meaning of concepts like "red" or "blue" by associating them with observed RGB values and visual features. This is done using weighted Kernel Density Estimation (wKDE).
  • Goal Inference: Learning the specific rules that constitute the task goal (e.g., "red blocks must be on blue blocks").
  • Handling Ambiguity: The model addresses ambiguity in teacher utterances (e.g., "red blocks on blue blocks" could refer to two different rules) by considering multiple interpretations and updating beliefs based on evidence.
  • Approximate Inference: Due to the increased complexity introduced by teacher fallibility, the paper employs approximate inference techniques (Bayesian Update with beam search) to manage the computational burden.

Datasets

Not discussed in the paper. The experiments use a simulated environment (blocks world) with a simulated teacher.

Key Insights and Ideas

  • The Teacher is Not Always Right: A fundamental insight is that ITL systems must be robust to human error. Assuming a perfect teacher is a significant limitation in real-world applications.
  • Observable vs. Latent Correction: The distinction between a teacher saying something and that utterance meaningfully correcting an action is key to handling fallibility. This allows the agent to infer the teacher's reliability.
  • Learning from Silence: The model implicitly learns from teacher silence, inferring that if no correction is given, the latest action likely adheres to the known rules.
  • Stabilizing Learning: Mistake-aware agents significantly reduce the variance in learning outcomes, preventing catastrophic failures that can occur when learning from a faulty teacher without this awareness.
  • Indirect Violations are Tricky: The paper suggests that missing indirect violations (where a mistake isn't immediately obvious but leads to an unrecoverable state later) are more common and pose a unique challenge.

Key Results

The paper presents experimental results comparing a "baseline agent" (BA) that assumes a faultless teacher with a "mistake-aware agent" (MA) that uses the proposed model.

  • Reduced Regret with Mistake Awareness: The MA agent consistently exhibits lower "regret" (number of mistakes made) compared to the BA agent, especially when the teacher makes direct violations.
  • Stabilized Performance: Figure 8 shows that the MA agent's performance is much more stable, with a narrower range of outcomes and fewer instances of extremely high regret, unlike the BA agent which can perform very poorly in some cases.
  • Learning Still Achieved: Despite teacher errors, the MA agent successfully learns the task and symbol grounding, demonstrating the effectiveness of the approach.
  • Indirect Violations Less Impactful: When teachers only miss indirect violations, the performance difference between BA and MA is less pronounced, suggesting that direct violations are more disruptive to learning without mistake awareness.

Limitations and Future Work

  • Teacher Fallibility Model: The paper explicitly sets the probabilities of teacher errors for experimental control. Future work could involve learning these probabilities from data.
  • Complexity of Errors: While four types of errors are considered, real-world human error can be more complex.
  • Generalization: The experiments are conducted in a blocks world. Future work could explore generalization to more complex environments and tasks.
  • Real-World Teachers: The current work relies on a simulated teacher. Testing with actual human teachers would be a crucial next step.

Summary updated

BibTeX
@inproceedings{appelgren2021symbol,
  author = {Mattias Appelgren and Alex Lascarides},
  title = {{Symbol Grounding and Task Learning from Imperfect Corrections}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure