Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systems

Vishal Sunder, Eric Fosler-Lussier, Samuel Thomas, Hong-Kwang Jeff Kuo, Brian Kingsbury

Abstract

End-to-End (E2E) Spoken Language Understanding (SLU) systems aim to map audio directly to intent, bypassing the traditional cascaded Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) pipeline. However, E2E models often struggle to capture the deep semantic nuances inherent in text-based models like BERT. This paper introduces a novel pretraining framework that aligns speech representations with BERT text embeddings on a token-by-token basis. By utilizing a cross-modal attention mechanism and a tokenwise contrastive loss, the proposed method distills semantic knowledge into a compact speech encoder. Experimental results on the SNIPS-Smartlights and Fluent Speech Commands datasets demonstrate that this fine-grained alignment achieves state-of-the-art performance while maintaining a small footprint suitable for on-device deployment.

Introduction

Modern voice assistants typically rely on a cascaded architecture for Spoken Language Understanding (SLU). In this setup, an ASR model first transcribes audio into text, which is then processed by an NLU model to extract intent. While effective, this approach has two primary drawbacks: high computational overhead and the risk of error propagation, where ASR inaccuracies lead to total NLU failure.

End-to-End (E2E) systems address these issues by mapping speech directly to semantics. However, bridging the gap between acoustic signals and high-level semantic meaning remains a challenge. Previous research has attempted to align speech and text at the sequence level (the entire utterance), but such coarse alignment often fails to capture the specific semantic weight of individual words. This paper proposes a transition from sequence-level to token-level alignment, allowing the model to learn more precise representations of spoken language.

Methodology

The proposed framework utilizes a teacher-student distillation approach where a frozen BERT model serves as the teacher for a compact speech encoder.

1. Speech Encoder Architecture

The student model is a 9-layer bidirectional LSTM (BiLSTM). To handle the high frame rate of audio data, the encoder employs a "pyramid" structure that reduces the frame rate by a factor of eight. This is followed by a self-attention layer to capture temporal dependencies.

2. Cross-Modal Attention

To achieve token-level alignment without explicit timing information (e.g., forced alignment), the authors implement a cross-modal attention mechanism. During pretraining, non-contextual word embeddings from a transcript act as the "query," while the speech encoder's outputs serve as the "keys" and "values." This mechanism automatically learns to identify which segments of the audio correspond to specific text tokens.

3. Tokenwise Contrastive Loss

The core innovation is the tokenwise contrastive loss. Instead of comparing global sentence embeddings, the model compares the representation of each individual token derived from the speech signal against its corresponding representation from the frozen BERT model. The loss function is designed to minimize the distance between "positive pairs" (the same token in speech and text) while maximizing the distance between "negative pairs" (different tokens).

4. Fine-tuning for Intent Recognition

Following pretraining on paired speech-text data, the model is fine-tuned for specific intent classification tasks using only audio. During this phase, the model utilizes the [CLS] token representation to predict the user's intent, effectively utilizing the semantic knowledge distilled during pretraining.

Experimental Setup

Datasets

  • Pretraining: The model was pretrained on the LibriSpeech dataset (960 hours of read English speech).
  • Evaluation: Performance was measured on two key SLU benchmarks:
  • SNIPS-Smartlights: Includes commands for smart home lighting in both "close-field" (clean) and "far-field" (noisy) environments.
  • Fluent Speech Commands (FSC): A large-scale dataset of common household automation commands.

Results and Discussion

Performance Benchmarks

The proposed model achieved state-of-the-art (SOTA) results across multiple tasks:

  • SNIPS-Smartlights: The model reached 92.23% accuracy in close-field settings and 83.07% in far-field settings, outperforming previous SOTA baselines by 3.5% and 8%, respectively.
  • Fluent Speech Commands: The model achieved 99.6% accuracy, matching the performance of significantly larger and more complex Transformer-based models.

The Importance of Granularity

A key finding of the ablation study was the superiority of token-level alignment. The tokenwise contrastive approach outperformed a sequence-level (pooled) contrastive approach by nearly 10% on the SNIPS close-field task. This confirms that fine-grained alignment is essential for the speech encoder to learn the semantic nuances required for accurate intent recognition.

Efficiency and Robustness

With only 48 million parameters, the model is highly efficient compared to massive Transformer architectures, making it ideal for edge deployment. Furthermore, the application of SpecAugment (masking frequency and time blocks) during fine-tuning significantly enhanced robustness, particularly in noisy far-field conditions.

Conclusion

This research demonstrates that tokenwise contrastive pretraining is a powerful method for aligning speech and text representations. By distilling BERT's semantic power into a compact BiLSTM encoder, the authors have created a system that is both highly accurate and computationally efficient. Future work will explore extending this framework to slot-filling tasks and investigating the use of more advanced speech encoders like HuBERT or wav2vec 2.0 to further enhance performance.

Summary updated

Abstract

Recent advances in End-to-End (E2E) Spoken Language Understanding (SLU) have been primarily due to effective pretraining of speech representations. One such pretraining paradigm is the distillation of semantic knowledge from state-of-the-art text-based models like BERT to speech encoder neural networks. This work is a step towards doing the same in a much more efficient and fine-grained manner where we align speech embeddings and BERT embeddings on a token-by-token basis. We introduce a simple yet novel technique that uses a cross-modal attention mechanism to extract token-level contextual embeddings from a speech encoder such that these can be directly compared and aligned with BERT based contextual embeddings. This alignment is performed using a novel tokenwise contrastive loss. Fine-tuning such a pretrained model to perform intent recognition using speech directly yields state-of-the-art performance on two widely used SLU datasets. Our model improves further when fine-tuned with additional regularization using SpecAugment especially when speech is noisy, giving an absolute improvement as high as 8% over previous results.
BibTeX
@inproceedings{sunder2022tokenwise,
  author = {Vishal Sunder and Eric Fosler-Lussier and Samuel Thomas and Hong-Kwang Jeff Kuo and Brian Kingsbury},
  title = {{Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systems}},
  booktitle = {Interspeech},
  year = {2022},
  doi = {10.21437/interspeech.2022-239},
}
Figure