Minimization of Regression and Ranking Losses with Shallow Neural Networks on Automatic Sincerity Evaluation

Hung-Shin Lee, Yu Tsao, Chi-Chun Lee, Hsin‐Min Wang, Wei-Cheng Lin, Wei‐Chen Chen, Shan-Wen Hsiao, Shyh‐Kang Jeng2016 · DOI

TL;DR

This paper introduces a novel framework for automatic sincerity evaluation that simultaneously optimizes both regression (predicting a precise sincerity score) and ranking (ordering utterances by sincerity) losses. By combining these objectives within shallow neural networks, the authors achieve significant improvements over existing methods, particularly in accurately ranking utterances.

What Problem Does This Paper Address?

Estimating the degree of sincerity conveyed by a speech utterance is a challenging task due to its subjective nature. While traditional approaches often treat this as a regression problem (predicting a specific score), the authors argue that human perception of sincerity is more aligned with relative ranking. Existing regression models, even those with low error, can produce poor rankings. Conversely, a good ranking system might not necessarily provide accurate absolute scores. This paper aims to bridge this gap by developing a model that excels at both predicting sincerity values and, crucially, ordering utterances according to their perceived sincerity, as measured by Spearman's rank correlation coefficient.

Methods

The core innovation lies in a unified objective function that combines regression and ranking losses. This is implemented using shallow neural networks with two distinct architectural realizations:

  • Sampling-based Regression and Ranking Neural Networks (spR2NNs): This approach trains the network using both individual utterances (for regression) and randomly sampled pairs of utterances (for ranking). The network learns to process these different input types through separate initial layers that are then combined in shared layers.
  • Anchor-based Regression and Ranking Neural Networks (acR2NNs): This method reinterprets ranking as a reference-comparing problem. It uses clustering (K-means) to identify representative "anchor" utterances. The network then learns to predict the relative distances between other utterances and these anchors, effectively learning a ranking without explicit random sampling of pairs.

Both architectures utilize a combined loss function that balances the regression loss (e.g., Mean Squared Error) and the pairwise ranking loss, along with regularization to prevent overfitting. The training process involves backpropagation to update network parameters.

Datasets

The experiments were conducted on speech material from the Sincerity sub-challenge of ComParE 2016. This dataset consists of 6,373 features extracted using the OpenSMILE toolbox. The training set contains 655 utterances from 22 speakers, and the test set has 256 utterances from 10 speakers. Due to the lack of a provided development set, the authors created one by selecting 6 speakers from the training data, ensuring a balanced gender ratio and proportional sample size to the test set.

Key Insights and Ideas

  • Ranking is Crucial for Sincerity: The paper strongly advocates for the importance of ranking in sincerity evaluation, arguing that humans often make decisions based on relative comparisons rather than absolute scores.
  • Simultaneous Optimization is Key: Combining regression and ranking losses within a single objective function is more effective than optimizing them separately or sequentially.
  • Anchor-based Ranking Offers Stability: The acR2NNs, which use fixed anchor points for ranking, demonstrate more stable training curves compared to the spR2NNs that rely on random sampling, potentially due to reduced variance.
  • Shallow Networks Suffice: The use of shallow neural networks with appropriate regularization proves effective, circumventing the small sample size problem common in this task.

Key Results

The proposed methods significantly outperform the baseline (a Support Vector Regressor).

  • On the development set, the acR2NNs achieved a relative improvement of 27.1% over the baseline.
  • On the test set, the acR2NNs achieved a relative improvement of 6.8% over the baseline.
  • The anchor-based approach (acR2NNs) generally yielded better results than the sampling-based approach (spR2NNs).

Limitations and Future Work

  • Limited Development Data: The authors acknowledge that the creation of a development set by selecting a subset of speakers might not fully capture the complexity of hyper-parameter tuning for neural networks, especially regarding initial weights and optimal training epochs.
  • Anchor Sample Size: For acR2NNs, the paper suggests that a shortage of training samples might limit the theoretical learning power of models with a larger number of anchor samples or more complex layer structures.
  • Future Work: The paper implicitly suggests further exploration into optimal anchor selection strategies and more sophisticated methods for hyper-parameter tuning, especially in scenarios with limited development data. Investigating the impact of different regularization techniques and network architectures could also be fruitful.

Summary updated

BibTeX
@misc{lee2016minimization,
  author = {Hung-Shin Lee and Yu Tsao and Chi-Chun Lee and Hsin‐Min Wang and Wei-Cheng Lin and Wei‐Chen Chen and Shan-Wen Hsiao and Shyh‐Kang Jeng},
  title = {{Minimization of Regression and Ranking Losses with Shallow Neural Networks on Automatic Sincerity Evaluation}},
  year = {2016},
  doi = {10.21437/interspeech.2016-756},
}
Figure