Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations

Jaeyeon Kim, Injune Hwang, Kyogu Lee

Abstract

We propose a framework to learn semantics from raw audio signals using two types of representations, encoding contextual and phonetic information respectively. Specifically, we introduce a speech-to-unit processing pipeline that captures two types of representations with different time resolutions. For the language model, we adopt a dual-channel architecture to incorporate both types of representation. We also present new training objectives, masked context reconstruction and masked context prediction, that push models to learn semantics effectively. Experiments on the sSIMI metric of Zero Resource Speech Benchmark 2021 and Fluent Speech Command dataset show our framework learns semantics better than models trained with only one type of representation.

Introduction

Spoken Language Modeling (SLM) aims to develop "textless" AI systems that understand language directly from audio, bypassing the need for intermediate text transcriptions. While current SLMs have achieved significant success in capturing phonetic nuances (how words sound) and syntax (how words are ordered), they often struggle with semantics—the underlying meaning of the speech.

Most existing models rely on a single type of speech representation, typically focused on fine-grained phonetic details. This research posits that a phonetic-only view is insufficient for high-level conceptual understanding. To address this, the authors propose a framework that processes speech through two simultaneous lenses: high-resolution phonetic units and low-resolution contextual units. This dual-representation approach allows the model to observe both the "trees" and the "forest" of spoken language.

Methodology

Dual-Channel Architecture

The core of the framework is a dual-channel Transformer architecture inspired by CharBERT, a model originally designed for character-level text processing. Unlike standard models that concatenate different data types into a single long sequence—which is computationally expensive and difficult for the model to parse—this architecture maintains two distinct, parallel channels.

Speech-to-Unit Processing

The pipeline begins by passing raw audio through two different pretrained models:

  1. Phonetic Representations: High-resolution units capturing fine-grained acoustic details.
  2. Contextual Representations: Lower-resolution units where each unit covers a longer time span, designed to capture broader word-level or phrase-level meanings.

To ensure these channels benefit from one another, a "Heterogeneous Interaction" module allows the channels to exchange information after every layer, fusing and then re-splitting data to enrich both representations.

Training Objectives

The model is trained using two novel tasks designed to force the learning of semantic relationships:

  • Masked Context Prediction (MCP): Similar to the Masked Language Modeling (MLM) used in BERT, the model must predict missing contextual units by analyzing the surrounding context and the corresponding phonetic units.
  • Masked Context Reconstruction (MCR): The model attempts to reconstruct the original continuous contextual features before they were quantized into discrete units. This task helps the model retain acoustic nuances that might otherwise be lost during the discretization process.

Experimental Results

Semantic Similarity (sSIMI)

The framework was evaluated using the Zero Resource Speech Benchmark 2021 sSIMI metric, which measures how well a model understands word meanings using the mturk-771 dataset. The proposed framework achieved an average sSIMI score of 12.97, a substantial improvement over the 9.32 achieved by a standard BERT-small model trained on a single representation.

Spoken Language Understanding (SLU)

To test intent classification, the researchers used the Fluent Speech Commands (FSC) dataset (e.g., classifying the command "turn on the lights"). The model achieved 96.76% accuracy on the standard test set. Notably, on the more challenging "unseen utterance" split, it reached 84.16%, significantly outperforming the baseline of approximately 74%.

Discussion and Key Insights

The Importance of Resolution

The results suggest that semantics are captured most effectively when a model is forced to align coarse-grained context with fine-grained phonetics. The dual-channel approach provides a more efficient and effective way to manage this alignment than joint-sequence methods.

Efficiency and Scaling

The model achieves state-of-the-art semantic performance with a relatively small parameter count (~27M) and a modest training budget of 60 GPU hours. This demonstrates that architectural innovation, specifically the integration of multi-resolution representations, can be more impactful than simply increasing model scale.

Limitations

The researchers noted a trade-off regarding the MCR task: while it significantly boosted performance on natural speech (LibriSpeech), it led to a slight decline in performance on synthesized (robotic) speech. This suggests the model may become highly tuned to the specific acoustic qualities of natural human voices, potentially limiting its generalization to artificial audio.

Conclusion

By incorporating both contextual and phonetic representations through a dual-channel architecture, this framework significantly advances the ability of textless Spoken Language Models to grasp semantic meaning. Future work may involve incorporating additional speech dimensions, such as prosody or emotion, and testing the framework's scalability on even larger datasets.

Summary updated

Abstract

We propose a framework to learn semantics from raw audio signals using two types of representations, encoding contextual and phonetic information respectively. Specifically, we introduce a speech-to-unit processing pipeline that captures two types of representations with different time resolutions. For the language model, we adopt a dual-channel architecture to incorporate both types of representation. We also present new training objectives, masked context reconstruction and masked context prediction, that push models to learn semantics effectively. Experiments on the sSIMI metric of Zero Resource Speech Benchmark 2021 and Fluent Speech Command dataset show our framework learns semantics better than models trained with only one type of representation.
BibTeX
@inproceedings{kim2024learning,
  author = {Jaeyeon Kim and Injune Hwang and Kyogu Lee},
  title = {{Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=z37xIhVXm0},
}
Figure