Prosodic Representations of Prominence Classification Neural Networks and Autoencoders Using Bottleneck Features

Sofoklis Kakouros, Antti Suni, Juraj Šimko, Martti VainioInterspeech 2019

TL;DR

This paper investigates how neural networks learn to represent "prominence" in speech, a key prosodic feature that signals emphasis. By analyzing the internal workings of both classification and autoencoder networks, the researchers found that these models learn distinct, yet informative, representations of prominence, with F0 (pitch) being a particularly strong acoustic correlate.

What Problem Does This Paper Address?

Prominence in speech is crucial for conveying meaning and emphasis, but its acoustic underpinnings are complex and involve an interplay of features like energy, fundamental frequency (F0), spectral tilt, and duration. Understanding how humans and machines perceive and represent prominence is challenging, especially given the difficulty in interpreting the internal "learned representations" of neural networks. This paper aims to shed light on what acoustic prosodic information these networks capture and how they organize it, particularly focusing on the compressed "bottleneck features" learned by autoencoders.

Methods

The researchers employed two main types of neural networks:

  • Prominence Classification Network: A standard feed-forward neural network trained to classify speech segments as either prominent or non-prominent. This network takes 55-dimensional word vectors (derived from statistical descriptors of acoustic features over a context of surrounding words) as input and outputs a binary prominence label.
  • Autoencoder Network: This network is trained to reconstruct its input. It compresses the 55-dimensional word vectors into a lower-dimensional "bottleneck" representation (specifically, 10-dimensional bottleneck features or BNFs) and then attempts to reconstruct the original input from this compressed representation. This is a common unsupervised approach to learn compact feature representations.

Both networks were trained on acoustic features (energy, F0, spectral tilt) extracted from Dutch continuous speech. To analyze the learned representations, they used k-nearest neighbor (kNN) classification on the network activations at different layers, testing their ability to distinguish between prominence, gender, and speaker labels. Visualization techniques, including Principal Component Analysis (PCA), were used to explore the high-dimensional feature spaces.

Datasets

The study utilized the Spoken Dutch Corpus (CGN), specifically the prosodically annotated subset of the Dutch news broadcast component (component k). This corpus contains approximately 7438 words from 134 news broadcasts spoken by 10 different speakers, with manual prominence annotations. For evaluation, a 10-fold cross-validation procedure was used, ensuring no speaker overlap between training and testing sets within each fold.

Key Insights and Ideas

  • F0 is King (for Prominence): Fundamental frequency (pitch) emerged as the most informative individual acoustic feature for prominence classification, outperforming energy and spectral tilt.
  • Combinations Help, But Not Infinitely: While combining acoustic features generally improved classification performance, the best performance was achieved with all three features (energy, F0, spectral tilt) together.
  • Networks Learn Distinct Representations: Both the classification network and the autoencoder learned representations that could distinguish prominence, but they did so differently. The classification network focused more directly on prominence, while the autoencoder also captured speaker and gender information more effectively in its representations.
  • Bottleneck Features are Informative: The 10-dimensional bottleneck features (BNFs) learned by the autoencoder were shown to provide good separation of prominence categories, demonstrating the power of dimensionality reduction for capturing essential prosodic information.
  • Speaker Variability is Significant: PCA visualizations revealed that a substantial portion of the variation in acoustic features is due to speaker-specific characteristics, suggesting that effective prominence detection requires models to either normalize or leverage this variability.

Key Results

  • Classification Performance: The supervised classification network achieved an accuracy of 77.53% for F0 alone and 82.50% when combining energy, F0, and spectral tilt.
  • Representation Discrimination (kNN):
    • The classification network showed increasing ability to discriminate prominence across hidden layers, reaching high accuracy (e.g., 88.35% for F0 in the 10-dimensional bottleneck layer). It also showed some ability to discriminate gender and speaker, though this decreased in later layers.
  • The autoencoder also learned representations that discriminated prominence, but its performance was generally lower than the classification network for prominence itself. However, it excelled at discriminating speaker labels (e.g., 89.86% for F0 in the bottleneck layer), outperforming the classification network on this task.
  • Visualization: PCA plots of the autoencoder's bottleneck features showed distinct clusters for prominent vs. non-prominent speech segments, with F0 yielding the clearest separation. Speaker-specific patterns were also evident in these visualizations.

Limitations and Future Work

  • Limited Acoustic Features: The study focused on only three acoustic features (energy, F0, spectral tilt) and did not include duration. Future work should incorporate more features, such as Mel-frequency cepstral coefficients (MFCCs) and duration.
  • Fixed Temporal Context: The analysis was performed over a fixed temporal context (word-level statistical descriptors with preceding/following word windows). Future research could explore different linguistic units (e.g., syllables) and more flexible temporal modeling.
  • Exploratory Nature: The study is described as exploratory, and the authors suggest extending experiments with larger datasets and investigating different modeling approaches for temporal context.

Summary updated

BibTeX
@inproceedings{kakouros2019prosodic,
  author = {Sofoklis Kakouros and Antti Suni and Juraj Šimko and Martti Vainio},
  title = {{Prosodic Representations of Prominence Classification Neural Networks and Autoencoders Using Bottleneck Features}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure