Tiny, Always-on, and Fragile: Bias Propagation through Design Choices in On-device Machine Learning Workflows

Wiebke Toussaint Hutiri, Aaron Yi Ding, Fahim Kawsar, Akhil MathurACM Trans. Softw. Eng. Methodol 2023 · DOI

TL;DR

This paper investigates how design choices in on-device machine learning (ML) workflows can introduce and amplify bias, leading to unreliable performance across different user groups. By analyzing audio keyword spotting (KWS) systems, the authors demonstrate that seemingly minor decisions during model training and optimization can significantly impact fairness, offering practical strategies for engineers to build more inclusive on-device ML.

What Problem Does This Paper Address?

As more ML models are deployed directly on resource-constrained devices (like smartphones and IoT gadgets) for privacy and speed, a critical challenge emerges: ensuring these "on-device ML" systems are not only accurate but also fair. Unlike cloud-based ML, on-device ML is highly sensitive to hardware limitations, user context, and a myriad of engineering decisions. This paper tackles the under-explored problem of how these design choices, made throughout the ML workflow, can lead to "reliability bias"—systematic performance disparities across user groups (e.g., based on gender). This bias can result in devices failing disproportionately for certain users, undermining the promise of accessible technology and potentially causing harm.

Methods

The authors adopt a software engineering lens, treating ML development as a workflow with distinct design choices. They introduce a "decision map" to visualize these choices. Their core methodology involves:

  • Quantifying Reliability Bias: They propose a parity-based measure to quantify performance disparities between user groups, focusing on accuracy metrics.
  • Factorial ANOVA: This statistical technique is used extensively to analyze the impact of various design choices (independent variables) on model accuracy and reliability bias (dependent variables), including their interactions.
  • Empirical Experiments: They conduct extensive experiments on an audio keyword spotting (KWS) task, a common on-device ML application.
  • Model Compression Analysis: They specifically investigate the effects of model pruning hyperparameters on bias.
  • Model Selection Strategies: They propose and evaluate multi-objective criteria for selecting models that balance accuracy and reliability bias.

Datasets

The study utilizes the following datasets:

  • Google Speech Commands (google_sc): An English language dataset for keyword spotting.
  • Multilingual Spoken Words Corpus (MSWC): This corpus was adapted to create four keyword spotting datasets in different languages:
  • MSWC English (mswc_en)
    • MSWC German (mswc_de)
    • MSWC French (mswc_fr)
    • MSWC Kinyarwanda (mswc_rw)

These datasets were carefully processed to ensure gender balance by matching utterances per gender.

Key Insights and Ideas

  • Design Choices are Bias Amplifiers: The paper strongly emphasizes that bias isn't just in the data; it's actively introduced and amplified by engineering decisions throughout the ML pipeline.
  • "Tiny" Doesn't Mean "Fair": Resource constraints in on-device ML don't excuse fairness considerations. In fact, the need to optimize for efficiency can exacerbate bias if not carefully managed.
  • Interactions Matter: It's not just individual design choices but their complex interactions that drive bias. For example, the impact of sample rate can be different depending on the model architecture.
  • Dataset Influence is Huge: The characteristics of the training data (language, size, speaker representation) significantly influence how design choices affect bias.
  • Pruning Can Worsen Bias: While model compression techniques like pruning reduce model size, they can also increase reliability bias, especially at high sparsity levels and with smaller learning rates.
  • Multi-Objective Model Selection is Key: Engineers should move beyond optimizing solely for accuracy and actively consider reliability bias when selecting models, especially for deployment.

Key Results

  • Sample Rate & Architecture: Lower sample rates (8kHz vs. 16kHz) and lighter architectures (like IICNN) generally lead to lower accuracy and higher reliability bias.
  • Pre-processing Parameters:
    • MFCC features generally perform better than raw log Mel spectrograms for accuracy.
  • Reducing the dimensionality of MFCCs (fewer filter banks/cepstral coefficients) can improve computational efficiency with minimal impact on accuracy and bias.
  • The choice of feature type and dimensions can significantly influence bias, sometimes favoring males and other times females, depending on the dataset.
  • Pruning Hyper-parameters:
    • Polynomial decay is a more robust pruning schedule than constant sparsity.
  • Larger pruning learning rates (e.g., 0.001) help mitigate bias and unexpected accuracy degradation, especially when pruning to high sparsities (>50%).
  • Pruning can increase reliability bias, particularly for smaller architectures and lower sample rates.
  • Model Selection: A "low bias + high accuracy" selection strategy can significantly reduce reliability bias with only a marginal drop in accuracy (within a 1.5% threshold), making it a practical approach for engineers.

Limitations and Future Work

  • Gender as the Sole Attribute: The study primarily focuses on gender bias. Future work should explore bias across other demographic attributes (age, ethnicity, etc.) and their intersections.
  • Binary Classification: The gender classification was binary, excluding non-binary individuals.
  • Limited Architectures: The study focused on CNN architectures. Future work should investigate bias in other model types.
  • Quantization Not Studied: The impact of quantization, another common model compression technique, on bias was not examined.
  • System Efficiency: While accuracy is analyzed, the impact of bias on system efficiency (e.g., power consumption, user attempts) is noted as an area for future research.
  • Other Modalities: The methodology is currently focused on audio ML; extending it to other modalities like images is suggested.
  • Dataset Representation: The study acknowledges that dataset representation is crucial and can explain performance deviations, especially for datasets with fewer speakers or utterances. Further research is needed on creating balanced datasets when data availability is variable.

Summary updated

BibTeX
@article{hutiri2023tiny,
  author = {Wiebke Toussaint Hutiri and Aaron Yi Ding and Fahim Kawsar and Akhil Mathur},
  title = {{Tiny, Always-on, and Fragile: Bias Propagation through Design Choices in On-device Machine Learning Workflows}},
  journal = {ACM Trans. Softw. Eng. Methodol},
  year = {2023},
  doi = {10.1145/3591867},
}
Figure