Non-linear Prediction of Acoustic Vectors Using Hierarchical Mixtures of Experts

Steve R. Waterhouse, Anthony J. RobinsonNeurIPS 1994

TL;DR

This paper demonstrates how Hierarchical Mixtures of Experts (HME) can significantly improve the prediction of acoustic vectors for speech coding. By using a "divide and conquer" architecture, the authors achieve higher prediction gains than traditional linear models and provide a framework for generating time-varying uncertainty estimates (error bars) that can be used to optimize data compression.

What Problem Does This Paper Address?

In the world of speech coding—the technology that allows us to transmit voice over digital networks—the goal is to represent speech with as few bits as possible without losing quality. A common technique is Linear Predictive Coding (LPC), which predicts the next segment of speech based on previous ones and only transmits the "error" (the difference between the prediction and reality).

However, speech is a highly non-linear and non-stationary signal. While standard linear models are fast, they struggle with the complex transitions in human speech. On the other hand, non-linear models like Multi-Layer Perceptrons (MLPs) are powerful but act as "black boxes"—they are slow to train and don't easily tell you how confident they are in a specific prediction. The authors seek a middle ground: a model that is non-linear, fast to train, and provides a statistical measure of its own uncertainty.

Methods

The authors employ the Hierarchical Mixture of Experts (HME). Think of this as a corporate hierarchy for data:

  1. Expert Networks: At the bottom are "experts" (in this case, simple linear models) that specialize in specific parts of the input space (e.g., one expert might handle vowels, another handles silence).
  2. Gating Networks: These act as managers. They look at the input and decide which expert’s opinion should carry the most weight for that specific moment.
  3. The Hierarchy: These are arranged in a tree structure, where managers report to higher-level managers until a single output is produced.
Hierarchical Mixture of Experts (HME) architecture showing gating networks (GN) and expert networks (EN) arranged in a tree structure.

What’s New? * Fast Optimization: Instead of standard backpropagation, they use the Expectation-Maximization (EM) algorithm combined with Iteratively Reweighted Least Squares (IRLS), making it an order of magnitude faster than MLPs. * Local Error Bars: The authors derive a way to calculate the variance (uncertainty) of the output. Because the HME is a statistical mixture, they can compute a "confidence interval" for every single prediction. * Dynamic Vector Quantization (VQ): They propose a scheme where the codebooks used to compress the error signal are mixed on-the-fly based on which "experts" are currently active.

Datasets

The paper evaluates the HME on three distinct levels:

  1. Toy Problem: Computer-generated data with known non-linearities and noise to prove the model can recover the underlying function and its variance.
  2. Sunspots: A classic time-series benchmark consisting of yearly sunspot activity from 1700 to 1979.
  3. Speech Data: The Resource Management (RM) corpus, using 100,000 frames for training and another 100,000 for evaluation. They specifically looked at 23 Mel-scale log energy frequency bins.

Key Insights and Ideas

  • Specialization is Key: The "expert activation plots" show that the model naturally learns to assign different experts to different acoustic features. For example, one expert might focus on the "lulls" in a signal while another handles the "peaks."
  • Uncertainty as a Tool: By calculating local error bars, the model doesn't just say "the next value is X"; it says "the next value is X, and I'm 95% sure it's within this range." In speech coding, this allows you to spend fewer bits when the model is confident and more bits when it isn't.
Performance on a toy problem demonstrating the model's ability to fit non-linear data and accurately estimate local variance (uncertainty).
  • Efficiency via Architecture: Because the experts are linear but the gating networks are non-linear, the HME captures complex behavior without the massive computational overhead of a fully connected deep neural network.

Key Results

  • Sunspot Prediction: The HME achieved a Normalized Mean Squared Error (NMSE) of 0.061 on training data, outperforming both the MLP (0.082) and the Threshold Autoregressive (TAR) model (0.097).
  • Speech Prediction Gain: In predicting Mel-scale frequency bins, a 2-level HME achieved a test prediction gain of 16.39 dB, a massive improvement over the 10.95 dB achieved by standard linear prediction.
  • Training Speed: The authors confirmed that the HME is at least an order of magnitude faster to train than standard MLPs on regression tasks.

Limitations and Future Work

  • Feature Sensitivity: While the model worked exceptionally well on Mel-scale energy bins, initial experiments on other common speech features like Line Spectral Pairs (LSPs) and cepstrals showed only "small improvements" over linear predictors. The authors note that further investigation is needed to understand why.
  • Variable Bit Rate Coding: The authors suggest that the next logical step is to build a "variable bit rate" coder. This would use the HME’s error bars to dynamically adjust how much data is sent, maintaining a constant level of speech quality while further reducing the total number of bits transmitted.

Summary updated

BibTeX
@inproceedings{waterhouse1994non,
  author = {Steve R. Waterhouse and Anthony J. Robinson},
  title = {{Non-linear Prediction of Acoustic Vectors Using Hierarchical Mixtures of Experts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1994},
}
Figure