``Alexa, can you forget me?'' Machine Unlearning Benchmark in Spoken Language Understanding

Alkis Koudounas, Claudio Savelli, Flavio Giobergia, Elena BaralisInterspeech 2025

TL;DR

This paper introduces UnSLU-BENCH, the first benchmark for machine unlearning in Spoken Language Understanding (SLU), addressing the critical need to remove specific user data from voice AI models. They evaluate eight unlearning techniques across four languages and propose a novel metric, GUM, to holistically assess efficacy, efficiency, and utility.

What Problem Does This Paper Address?

As AI systems, particularly voice assistants like Alexa, collect vast amounts of user speech data, ensuring user privacy and compliance with regulations like GDPR (the "right to be forgotten") becomes paramount. Machine unlearning (MU) offers a way to remove specific data points from trained models without complete retraining. However, while MU has been explored in text and image domains, its application to complex SLU tasks, which involve understanding intent and context from speech, remains underdeveloped. This paper aims to bridge this gap by providing a standardized benchmark and evaluation framework for MU in SLU.

Methods

The authors evaluate eight existing machine unlearning techniques:

  • Fine-Tuning (FT): Standard retraining on the retained data for a short period.
  • Negative Gradients (NG): Reverses gradients during backpropagation on the forget set to "unlearn" it.
  • NegGrad+ (NG+): Extends NG by also fine-tuning on the retain set to mitigate catastrophic forgetting.
  • Catastrophically forgetting the last k layers (CF-k): Applies fine-tuning only to the final layers of the model for efficiency.
  • UNSIR: A two-phase method involving an "impair" phase (error-maximizing noise) and a "repair" phase (fine-tuning).
  • Bad Teaching (BT) & BT-L: Uses a "teacher-student" setup with a competent and an incompetent teacher to guide the unlearning process.
  • SCRUB: Another teacher-student method that uses a single teacher and combines losses to maximize similarity on retained data and minimize it on forgotten data.

A key methodological contribution is the Global Unlearning Metric (GUM). This metric combines three crucial aspects:

  • Utility (U): How well the unlearned model performs the original task (measured by F1 score similarity to the gold model).
  • Efficacy (E): How well the model has forgotten the specific data (measured using Membership Inference Attacks (MIA), comparing the unlearned model's MIA to the original and gold model's MIA).
  • Efficiency (T): The computational cost of the unlearning process (measured by speedup relative to retraining).

GUM is a weighted harmonic mean of these three components, allowing for a balanced evaluation.

Datasets

The benchmark, UnSLU-BENCH, is built upon four publicly available SLU datasets spanning four languages:

  • Fluent Speech Commands (FSC): English, 31 intents.
  • SLURP: English, 60 intents. (The authors created new speaker-independent splits, denoted as SLURP*).
  • ITALIC: Italian, 60 intents.
  • SpeechMASSIVE: German and French, 60 intents each.

The models evaluated include wav2vec 2.0 and HuBERT for English datasets, and XLS-R 128 and XLS-R 53 (ASR-fine-tuned) for multilingual datasets.

Key Insights and Ideas

  • The "Right to be Forgotten" is Complex: Simply removing data from a database isn't enough; the model's learned parameters must also be purged of its influence.
  • No Single "Best" Unlearning Method: Different methods excel in different aspects (efficacy, speed, utility), making a holistic evaluation crucial.
  • Efficiency Matters: While many prior works focus on efficacy and utility, this paper highlights that practical unlearning requires significant speedups, which NG provides exceptionally well.
  • Catastrophic Forgetting is a Real Risk: Some methods, like NG+, can inadvertently degrade model performance on the retained data.
  • Training Duration Impacts Unlearning: Shorter training times might lead to models that are easier to unlearn from, while prolonged training can create more rigid, harder-to-forget patterns.

Key Results

  • Negative Gradients (NG) Shines: NG consistently achieved the highest GUM scores across most datasets and models, largely due to its exceptional efficiency (speedups up to 1748x) and strong efficacy.
  • GUM vs. Other Metrics: The paper demonstrates that metrics like NoMUS can be misleading by not properly contextualizing MIA scores relative to the gold and original models. GUM provides a more balanced view.
  • Trade-offs are Evident: For instance, FT can balance utility and efficacy but is less efficient. CF-k is efficient but risks incomplete unlearning.
  • Training Length Matters: For the NG+ method on SLURP*, an optimal training duration around 11 epochs was found to balance utility and memorization risk, before overfitting became detrimental to unlearning.

Limitations and Future Work

  • Gold Model Dependency: The GUM metric, and indeed the evaluation of unlearning, relies on a "gold model" (retrained from scratch). This is a limitation as gold models are not available in real-world deployment scenarios. The authors acknowledge that developing gold-free solutions is a future research direction.
  • Speaker-Independent Splits: While new splits were created for SLURP*, the availability of such splits for all SLU datasets would be beneficial for consistent evaluation.
  • Further Research Needed: The paper calls for continued research into more robust and trustworthy privacy-preserving techniques for speech AI systems. Future work could explore more advanced unlearning algorithms tailored for SLU and investigate unlearning in more complex scenarios, such as continual learning or federated learning.

Summary updated

BibTeX
@inproceedings{koudounas2025alexa,
  author = {Alkis Koudounas and Claudio Savelli and Flavio Giobergia and Elena Baralis},
  title = {{``Alexa, can you forget me?'' Machine Unlearning Benchmark in Spoken Language Understanding}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure