Likability classification - a not so deep neural network approach

Raymond Brueckner, Björn SchullerInterspeech 2012

TL;DR

This paper explores the effectiveness of Deep Belief Networks (DBNs) for classifying speaker likability. Surprisingly, they found that a simpler, two-layer neural network with a Gaussian-Bernoulli Restricted Boltzmann Machine (GBRBM) as the first layer significantly outperformed more complex DBNs and baseline methods, achieving a relative improvement of 8.5%.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically predicting a speaker's "likability" from their voice. This is a crucial aspect of human-computer and human-human interaction, impacting how users perceive and engage with voice-based systems like voice portals or social networks. The task involves analyzing acoustic speech signals to infer this subjective trait, which is a complex pattern recognition problem. The authors investigate whether advanced deep learning architectures, specifically DBNs, can effectively model the intricate acoustic patterns associated with likability, aiming to surpass existing baseline methods.

Methods

The core of this research lies in the application and comparison of different neural network architectures. The authors experiment with:

  • Restricted Boltzmann Machines (RBMs): These are fundamental building blocks for DBNs. They are generative models that learn probability distributions of input data in an unsupervised manner. The paper details both binary RBMs and Gaussian-Bernoulli RBMs (GBRBMs) for handling real-valued input data.
  • Deep Belief Networks (DBNs): DBNs are constructed by stacking multiple RBM layers. The idea is that each layer learns increasingly complex representations of the input data. The paper describes a greedy, layer-wise unsupervised pre-training approach followed by discriminative fine-tuning.
  • Two-Layer Neural Network with GBRBM Pre-training: This is a simpler architecture where a GBRBM is trained first on the input features, and its hidden layer outputs are then used as input to a second layer (a logistic regression classifier in this case) which is fine-tuned.
  • Baseline Methods: Support Vector Machines (SVMs) and Random Forests are mentioned as baseline approaches, though their specific implementation details are not elaborated upon in the experimental setup.

The training process involves stochastic gradient descent (SGD) on mini-batches, contrastive divergence for RBM pre-training, and cross-entropy cost for fine-tuning. Regularization techniques like weight decay and sparsity thresholds are also employed.

Datasets

The experiments were conducted on the "Speaker Likability Database" [9], which was used for the Likability Sub-Challenge of the Interspeech 2012 Speaker Trait Challenge. The speech data consists of recordings from fixed and mobile telephone lines, sampled at 8 kHz. The feature set comprises 6125 acoustic features derived from low-level descriptors and functionals.

Key Insights and Ideas

  • DBNs Don't Always Win: The most striking finding is that the complex, multi-layer DBNs did not yield the best results for this specific likability classification task. This suggests that for certain problems, the added complexity of deep generative models might not translate to improved discriminative performance.
  • The Power of a Single Layer: A surprisingly effective approach was a two-layer network where the first layer was a GBRBM. This unsupervised pre-training step, followed by fine-tuning, significantly boosted performance. This highlights the value of learning good feature representations from the data, even without explicit labels.
  • Unsupervised Pre-training is Key: The paper strongly emphasizes the benefit of unsupervised pre-training for learning meaningful features. These learned features are more robust and capture underlying statistical structures better than raw input features, even if some learned features are irrelevant for the final task.
  • GBRBMs for Real-Valued Data: The use of GBRBMs proved effective for the preprocessed, approximately Gaussian-distributed input features, demonstrating their suitability for real-valued acoustic data.

Key Results

The paper reports the following key quantitative results:

  • Baseline Performance: The random forest baseline achieved an Unweighted Accuracy (UA) of 59.0% on the test set.
  • MLP Performance: A standard one-hidden layer MLP with random initialization achieved 60.9% UA.
  • Best Performing Model: The two-layer neural network with a GBRBM pre-trained first layer achieved 64.0% UA on the test set. This represents a relative improvement of 8.5% over the baseline.
  • DBN Performance: More complex DBNs (1 to 4 layers) showed varying performance, with a 1-layer DBN reaching 64.0% UA, but performance generally decreased with more layers. For instance, a 4-layer DBN achieved 62.2% UA.
  • Bottleneck Architectures: Experiments with bottleneck DBN architectures (decreasing layer sizes) did not outperform the simpler models, with some even performing worse than the baseline.

Limitations and Future Work

  • Limitations: The authors acknowledge that the task of likability classification is inherently subjective and complex. They also note that their DBNs did not fully leverage their modeling power for this specific problem. The performance of DBNs decreased with added layers, suggesting potential overfitting or issues with the fine-tuning process for deeper architectures.
  • Future Work: The authors suggest exploring second-order optimization methods (like Conjugate Gradient or Quasi-Newton) for fine-tuning to improve efficiency. They also propose investigating the use of radial basis functions as an alternative to GBRBMs for the first layer, given the Gaussian nature of the preprocessed data. Finally, they express interest in exploring semi-supervised techniques for future work in emotion and speaker trait recognition.

Summary updated

BibTeX
@inproceedings{brueckner2012likability,
  author = {Raymond Brueckner and Björn Schuller},
  title = {{Likability classification - a not so deep neural network approach}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure