Feedback Attribution for Counterfactual Bandit Learning in Multi-Domain Spoken Language Understanding

Tobias Falke, Patrick LehnenEMNLP 2021 · DOI

Introduction: The Feedback Attribution Problem

Training modular AI systems using only user feedback (e.g., "no, that's wrong") is notoriously difficult because it is often unclear which specific component in the pipeline made the mistake. In modern voice assistants, the architecture is typically modular: a Domain Classifier (DC) first determines the domain (e.g., music vs. alarms), followed by domain-specific models for Intent Classification (IC) and Slot Labeling (SL).

The modular SLU pipeline and the feedback attribution (FA) process.

When a user provides feedback—either explicitly ("Stop, that's wrong") or implicitly (interrupting the device)—it is usually "bandit feedback." We know the entire prediction failed, but we don't know why. Did the DC pick the wrong domain? Did the IC misunderstand the intent? Or did the SL misidentify a date? If every model in the chain is penalized for every failure, the models that performed correctly are effectively "poisoned." This paper introduces feedback attribution (FA) methods to mathematically distribute blame across a multi-domain Spoken Language Understanding (SLU) pipeline, allowing models to improve from historical logs without expensive manual re-labeling.

Methods

The authors propose and compare two main strategies for attributing feedback ($\delta$):

  1. Propensity-based FA: This method assumes that if a model was "unsure" (low probability for its choice), it is more likely to be the culprit. It distributes negative feedback proportionally to the model's uncertainty.
  2. Advantage-based FA: Inspired by multi-agent reinforcement learning (specifically the COMA algorithm), this method uses a separate "feedback estimator" model. It asks a counterfactual question: "How much better (or worse) was this specific prediction compared to the average of all other possible predictions we could have made?" If the chosen prediction was significantly worse than the alternatives, it receives a higher penalty.

These methods are integrated into an Inverse Propensity Scoring (IPS) framework, enabling the system to estimate how a new policy would perform based on the logs of an old one.

Experimental Setup

The researchers evaluated these methods using two popular SLU benchmarks modified for multi-domain environments:

  • SNIPS: 14.5k utterances grouped into 3 domains (e.g., "BookRestaurant" and "GetWeather").
  • TOP (Task-Oriented Parsing): 43.3k utterances focused on complex navigation and event queries, grouped into 3 domains (Alarms, Reminders, Weather).

Initial "logging" models were trained on 5% of the labeled data, while the remaining 95% was used to generate "bandit logs" where only binary success/failure feedback was available.

Key Results

Performance was measured using Semantic Accuracy (Sem-Acc), which requires the domain, intent, and all slots to be correct.

Performance comparison of various feedback attribution methods on SNIPS and TOP datasets.

As shown in the results table, the "Overall" trap—applying negative feedback to all modules—leads to disastrous performance, often dropping accuracy 10-20 points below the starting point. Propensity-based attribution also performed poorly because neural networks are often confidently wrong; if a model doesn't "blame" itself, the error remains uncorrected.

In contrast, the Advantage-based (Adva-Uni) method significantly outperformed baselines:

  • SNIPS: The logging policy started at 57.87% Sem-Acc. Using Adva-Uni with sampled logs boosted this to 81.77% (Oracle: 94.53%).
  • TOP: The logging policy started at 76.07%. Adva-Uni improved this to 89.55%, nearly matching the Oracle's 92.02%.

Discussion and Limitations

The study highlights that while training on successful interactions ("Positive Only") is a strong baseline, Advantage-based methods can outperform it by effectively learning from mistakes. However, several limitations remain:

  • Simulated Feedback: The study uses perfect simulated feedback; real-world user feedback is often noisy or missing.
  • Exploration Requirements: These methods rely on the logging policy having some level of stochasticity (exploration). Deterministic production systems make counterfactual learning significantly harder.
  • Scope: Future work could extend this attribution logic beyond SLU to Dialogue Management or Natural Language Generation.

Summary updated

BibTeX
@inproceedings{falke2021feedback,
  author = {Tobias Falke and Patrick Lehnen},
  title = {{Feedback Attribution for Counterfactual Bandit Learning in Multi-Domain Spoken Language Understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.91},
}
Figure