Acoustic correlates of meaning structure in conversational speech

Alexei V. Ivanov, Giuseppe Riccardi, S. Ghosh, S. Tonelli, E. A. StepanovInterspeech 2010

TL;DR

This paper explores how acoustic features like pitch, intensity, and formant trajectories can directly reveal the underlying meaning structure of spoken language, bypassing traditional ASR and parsing methods. They demonstrate statistically significant correlations between these acoustic cues and semantic roles within the FrameNet framework, paving the way for more robust and semantically aware spoken language understanding systems.

What Problem Does This Paper Address?

Current Spoken Language Understanding (SLU) systems typically rely on a pipeline: Automatic Speech Recognition (ASR) to get word hypotheses, followed by parsing to extract meaning structures. This approach is hampered by high word error rates from ASR and the often ad-hoc nature of conceptual representations. The authors propose a more direct approach: can we infer meaning structure directly from the acoustic and non-verbal linguistic features of speech, without relying on transcribed words? They aim to discover "meaning components" – specifically, the most semantically important words (target words) and their associated roles (frame elements) – by analyzing the acoustic signal itself. This is motivated by research in language acquisition and computational linguistics, which suggests prosody plays a crucial role in conveying meaning.

Methods

The core of the paper is the investigation of acoustic features as direct indicators of semantic roles. They use the FrameNet semantic model, which defines situations (frames), the words that evoke them (lexical units/target words), and the participants in those situations (frame elements).

  1. FrameNet Annotation: They utilize the LUNA spoken dialog corpus, manually annotated with FrameNet's Predicate-Argument Structure (PAS). This annotation identifies target words and their associated frame elements.
  2. Acoustic Feature Extraction: They extract several acoustic features from the audio recordings:
  3. Pitch (F0): Fundamental frequency.
  4. Formant Trajectories (F1, F2, F3): Resonances of the vocal tract, related to vowel quality.
  5. Intensity (Itot): Overall loudness.
  6. Harmonicity (Ihnr): A measure related to the clarity of vocal fold vibration. They combine intensity and harmonicity to derive Iharm, aiming to capture phonation and paralinguistic stress more robustly.
  7. Feature Analysis: They analyze these acoustic features in relation to the annotated semantic roles (target words vs. frame elements vs. non-target words). This involves comparing absolute feature values to average values within a turn, looking for statistically significant differences.
  8. Classification: They also perform classification experiments to predict whether a token is a target word using:
  9. Lexical Features: Part-of-Speech (PoS) tags, lemmas, and token context.
  10. Acoustic Features: Using MLP and SVM classifiers.
  11. Combined Features: Integrating both lexical and acoustic information.

Datasets

  • LUNA spoken dialog corpus: This corpus consists of Italian human-human dyadic conversations in a problem-solving task (software/hardware troubleshooting). It was manually annotated with attribute-value information, Predicate-Argument Structure (PAS) based on FrameNet, and dialog acts. The paper reports statistics for training, development, and test sets derived from this corpus.

Key Insights and Ideas

  • Acoustic Cues for Meaning: The paper strongly suggests that acoustic features are not just carriers of phonetic information but also carry direct signals about the semantic role of words in an utterance.
  • Effort and Frequency Codes: The findings align with theories of prosody, specifically the "effort code" (e.g., louder, more intense speech for emphasis) and "frequency code" (e.g., lower pitch for dominance). For instance, target words show higher harmonic intensity margins (effort) and lower pitch deviations (frequency code).
  • Harmonic Intensity as a Stress Indicator: The harmonic component of speech intensity appears to be a more reliable indicator of paralinguistic stress than overall intensity, showing a much larger difference between target words and other segments.
  • Formant Dynamics and Effort: Larger formant frequency differences (F2, F3) for target words are linked to the "effort code," suggesting articulatory effort.
  • Potential for Direct Semantic Grounding: The ability to correlate acoustic features with semantic roles opens up possibilities for grounding meaning directly in the acoustic signal, potentially bypassing the need for explicit linguistic transcription in some SLU tasks.

Key Results

  • Statistically Significant Correlations: The study found statistically significant differences in acoustic features (pitch, intensity, formants) between target words, frame elements, and non-target words across the LUNA corpus.
  • Harmonic Intensity Margin: Target words exhibited an 8 dB higher maximal-to-mean harmonic intensity margin compared to other words, while the margin for total intensity was only 1.5 dB.
  • Pitch and Formant Dynamics: Target words showed lower average pitch deviations and larger average inter-frame formant frequency deltas compared to non-target words and frame elements.
  • Classification Performance:
    • Lexical features alone achieved an F1-score of 0.810 for target word classification.
  • Acoustic features alone achieved an F1-score of 0.375 (MLP) and 0.3397 (SVM), indicating that while acoustic cues are informative, they are less powerful than lexical features in isolation for this specific task.
  • The combination of lexical and acoustic features yielded the highest performance, with an oracle combination reaching an F1-score of 0.924.

Limitations and Future Work

  • Acoustic Classifier Performance: The authors acknowledge that acoustic classifiers alone did not achieve high performance (F1 ≈ 0.375) compared to lexical classifiers (F1 ≈ 0.810). They suggest that better employment of turn context might improve recognition performance.
  • Integration with Linguistic Models: They propose integrating acoustic classification results into linguistic models for automated language acquisition, suggesting that consistently identified acoustic cues for semantic roles could bootstrap linguistic understanding.
  • Contextualization of Formant Features: The authors note that formant features might be more informative if conditioned on the specific phoneme being uttered.
  • Voicing Duration: While target words showed longer voiced intervals, the authors were unsure if this was an independent feature or a byproduct of increased harmonic component intensity. Further investigation is needed.
  • Future Applications: The findings are seen as encouraging for applications in automated language acquisition and for building more robust SLU systems by integrating acoustic and lexical information.

Summary updated

BibTeX
@inproceedings{ivanov2010acoustic,
  author = {Alexei V. Ivanov and Giuseppe Riccardi and S. Ghosh and S. Tonelli and E. A. Stepanov},
  title = {{Acoustic correlates of meaning structure in conversational speech}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure