One-vs-All Models for Asynchronous Training: An Empirical Analysis

Rahul Gupta, Aman Alok, Shankar AnanthakrishnanInterspeech 2019

TL;DR

This paper investigates the impact of asynchronous training on One-vs-All (OVA) classification systems, a common approach in large-scale machine learning. They introduce a novel metric to quantify dataset divergence in OVA models and demonstrate its strong correlation with performance degradation, offering a way to monitor system health.

What Problem Does This Paper Address?

In machine learning, particularly for tasks with many categories (like voice assistants understanding commands), classifying an input can be done using a single multi-class model or by training multiple independent "One-vs-All" (OVA) models. Each OVA model handles one category, deciding if the input belongs to that category or not. A key advantage of OVA systems is that individual models can be updated asynchronously – meaning one model can be retrained without affecting others. This is crucial for large, evolving systems where constant updates are needed. However, this asynchronous training can lead to differences in the data used by each OVA model, potentially impacting overall accuracy. This paper aims to empirically analyze this impact and propose a way to measure and understand the "asynchrony" introduced by independent model updates.

Methods

The core of this paper is an empirical analysis of OVA systems under asynchronous training conditions. They introduce a novel metric, denoted by $\alpha$, to quantify the divergence between datasets used for training individual OVA models. This metric is computed by modeling the distribution of sentence embeddings (derived from word embeddings) of a dataset and its copies using Kernel Density Estimation (KDE). The $\alpha$ metric essentially measures how different the latent representations of utterances are across these datasets.

They then conduct two main experimental setups:

  1. Synthetic Experiments: They simulate asynchronous training on a domain classification task (a component of Spoken Language Understanding - SLU). They systematically vary three factors:
  2. Asynchrony Factor: By progressively reducing the proportion of "negative" samples (data from other classes) used to train a specific OVA model.
  3. Number of Data Points: By increasing the overall size of the training datasets.
  4. Number of Classes: By merging existing classes to create systems with fewer, larger classes. In these simulations, they compare the performance gap between the asynchronous OVA system and a synchronized multi-class system.

  5. Real-World SLU System Analysis: They apply their analysis to a large-scale, multi-component SLU system. Here, asynchrony is introduced more naturally by using older versions of data from other domains for training. They measure performance using the Semantic Error Rate (SemER).

Datasets

The paper uses two primary data sources:

  • Domain Classification Dataset: A synthetic dataset derived from approximately 1.5 million utterances directed towards Alexa devices, annotated with one of 23 domains. This dataset is used for the simulated experiments.
  • Spoken Language Understanding (SLU) System Data: A real-world SLU system dataset, also derived from Alexa requests, consisting of 23 domains. This dataset is used for the more naturalistic analysis of asynchronous training.

Key Insights and Ideas

  • Asynchrony has a tangible impact: The paper clearly demonstrates that asynchronous training in OVA systems, while offering flexibility, can lead to a performance degradation compared to synchronized multi-class systems.
  • Non-linear performance degradation: The performance gap between asynchronous OVA and synchronized multi-class systems tends to increase non-linearly, especially as the degree of asynchrony or the number of classes grows beyond certain thresholds.
  • Larger datasets mitigate asynchrony: Increasing the overall size of the training data helps to reduce the negative impact of asynchronous training, as random sub-sampling becomes less likely to significantly alter the underlying data distribution.
  • A useful health metric for OVA systems: The proposed asynchrony metric $\alpha$ shows a strong correlation with performance degradation. This suggests it can serve as a valuable tool for monitoring the "health" of an OVA system and identifying when re-synchronization or retraining might be necessary.
  • Real-world asynchrony is manageable: The study shows that even in a complex, real-world SLU system, the proposed metric and the observed trends hold, indicating the practical applicability of their findings.

Key Results

  • Synthetic Experiments:
    • The performance gap between OVA and multi-class systems increased significantly when more than 10% of negative samples were deleted.
  • Performance differences accelerated when the number of classes exceeded 8.
  • The asynchrony metric $\alpha$ showed strong correlations with performance gaps: 0.87 for asynchrony factor, 0.88 for dataset size, and 0.97 for the number of classes.
  • Real-World SLU System:
    • Relative degradation in Semantic Error Rate (SemER) increased rapidly beyond 4 months of data staleness.
  • The asynchrony metric $\alpha$ correlated well with SemER degradation (0.84), reinforcing its utility.

Limitations and Future Work

  • Limitations: The paper acknowledges that the synthetic experiments are simulations and may not perfectly capture all nuances of real-world systems. While the real-world SLU system analysis is more naturalistic, the degree of asynchrony is still controlled to some extent.
  • Future Work:
    • Fine-tuning the $\alpha$ metric: Exploring transformations on sentence embeddings before PDF estimation to potentially enhance the metric's sensitivity to dataset differences.
  • Gating asynchrony: Developing mechanisms to use the $\alpha$ metric to intelligently gate asynchrony and signal when model retraining is needed.
  • Theoretical grounding: Deriving a theoretical relationship between the $\alpha$ metric and model performance, beyond empirical observations.
  • Broader applications: Extending the analysis to other domains beyond Natural Language Understanding, such as sentiment analysis, emotion recognition, and language identification.

Summary updated

BibTeX
@inproceedings{gupta2019one,
  author = {Rahul Gupta and Aman Alok and Shankar Ananthakrishnan},
  title = {{One-vs-All Models for Asynchronous Training: An Empirical Analysis}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure