A Theory of Unsupervised Speech Recognition

Liming Wang, Mark A. Hasegawa-Johnson, Chang D. YooarXiv 2023 · DOI · arXiv

Abstract

Unsupervised Automatic Speech Recognition (ASR-U) has achieved significant empirical success, yet it remains a poorly understood "black box" characterized by training instability and sensitivity to hyperparameters. This paper establishes a rigorous theoretical framework for ASR-U by treating it as a decipherment problem. Leveraging Random Matrix Theory (RMT) and Neural Tangent Kernels (NTK), we identify the mathematical conditions necessary for learnability, specifically highlighting a "phase transition" in model performance dictated by the complexity of the language's transition patterns.

Introduction

Traditional Automatic Speech Recognition (ASR) systems rely on vast quantities of paired audio-text data. In contrast, Unsupervised ASR (ASR-U) seeks to bridge the gap between speech and text using only unpaired datasets. While models such as wav2vec-U have demonstrated that this is possible, they are notoriously difficult to train, often requiring exhaustive hyperparameter searches and being prone to convergence at suboptimal local minima.

Until now, the field lacked a formal explanation for why ASR-U works or under what conditions it is guaranteed to fail. This research provides the first comprehensive theoretical lens to analyze the learnability of speech from unpaired data, transforming ASR-U from a series of empirical heuristics into a mathematically grounded discipline.

Theoretical Framework: Decipherment and Dynamics

The authors frame ASR-U as a decipherment problem where the goal is to map a sequence of speech units to a sequence of text units without a parallel corpus. This relationship is modeled using a Hidden Markov Model (HMM) and analyzed through two primary mathematical tools:

1. Random Matrix Theory (RMT)

RMT is employed to quantify the "complexity" of a language. By examining the transition probability matrix—which dictates the likelihood of one phoneme following another—the researchers demonstrate that the diversity of the matrix's eigenvalues is a critical determinant of decipherability. If the eigenvalues are sufficiently distinct, the language contains enough statistical "landmarks" for a model to uniquely identify the mapping between speech and text.

2. Neural Tangent Kernels (NTK)

To understand the training dynamics of the Generative Adversarial Networks (GANs) typically used in ASR-U, the authors utilize NTK theory. By analyzing the "infinite-width" limit of these networks, they describe the training process as a system of differential equations. This allows for a formal proof of the conditions under which a GAN will converge to the correct translation rather than collapsing into a bad local optimum.

Experimental Methodology

To validate the theory, the researchers utilized six synthetic languages designed with specific mathematical properties. Unlike standard speech datasets, which are often too noisy for clean theoretical verification, these synthetic languages were based on:

  • Circulant graphs: Representing simple, highly symmetric structures.
  • De Bruijn graphs: Representing complex structures common in genomic sequences.
  • Hypercubes: Representing highly regular, symmetric structures.

By manipulating the "eigenvalue gaps" within these structures, the authors could precisely observe the threshold at which a language becomes learnable.

Key Insights: The Phase Transition of Learnability

The most significant finding is that ASR-U performance does not improve linearly with data or model size. Instead, it exhibits a phase transition.

  • The Complexity Threshold: If a language is too symmetric or simple (low eigenvalue diversity), the model fails to learn anything. However, once the language's transition matrix reaches a specific complexity threshold, the Phoneme Error Rate (PER) drops precipitously toward zero.
  • Symmetry as an Obstacle: High symmetry in a language makes different sounds statistically indistinguishable. Complexity provides the necessary "asymmetry" that allows the model to navigate the linguistic landscape.
  • Stability through Weight Resetting: A practical takeaway from the NTK analysis is that the "memory" of a GAN's discriminator can hinder progress. Periodically resetting the discriminator weights was found to significantly stabilize training, preventing the model from becoming trapped.
  • Gradient Averaging: The theory suggested that "outside cost" (averaging gradients) is more stable than feeding raw probabilities ("soft input"), a prediction confirmed by experimental results.

Results and Implications

The study successfully provided the first mathematical proof of the necessary and sufficient conditions for ASR-U learnability. Empirical tests on synthetic languages confirmed the predicted "tipping point" behavior. For instance, in tests involving Circulant graphs, the model consistently failed until the number of distinct eigenvalues exceeded the number of speech units. Furthermore, the theory-inspired modifications—such as discriminator resetting—yielded more stable training and lower error rates than standard state-of-the-art configurations used in models like wav2vec-U.

Limitations and Future Directions

While this framework represents a landmark achievement, several challenges remain for real-world application:

  • Quantization: The current theory assumes speech is pre-processed into discrete units. Real-world audio is continuous, adding a layer of signal processing complexity.
  • Segmentation: The model assumes known boundaries between phonemes, whereas segmenting continuous speech remains a significant hurdle in practice.
  • Asynchrony: The theory does not yet account for the differing temporal rhythms between speech and text modalities.

Future research will focus on extending this framework to continuous speech features, aiming for a "Grand Unified Theory" of unsupervised auditory learning.

Summary updated

Abstract

Unsupervised speech recognition (ASR-U) is the problem of learning automatic speech recognition (ASR) systems from unpaired speech-only and text-only corpora. While various algorithms exist to solve this problem, a theoretical framework is missing from studying their properties and addressing such issues as sensitivity to hyperparameters and training instability. In this paper, we proposed a general theoretical framework to study the properties of ASR-U systems based on random matrix theory and the theory of neural tangent kernels. Such a framework allows us to prove various learnability conditions and sample complexity bounds of ASR-U. Extensive ASR-U experiments on synthetic languages with three classes of transition graphs provide strong empirical evidence for our theory (code available at cactuswiththoughts/UnsupASRTheory.git).
BibTeX
@misc{wang2023theory,
  author = {Liming Wang and Mark A. Hasegawa-Johnson and Chang D. Yoo},
  title = {{A Theory of Unsupervised Speech Recognition}},
  year = {2023},
  eprint = {2306.07926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.07926},
}
Figure