Seal: Advancing Speech Language Models to be Few-Shot Learners

Shuyu Lei, Lingen Liu, Jiaolong Yang, Yasen Jiao, Yuxiang Yang, Yushu Yang, Xiang GuoarXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) like GPT-4 and Phi-3 have demonstrated remarkable few-shot learning capabilities in the text domain. However, these models remain "deaf" to direct audio input. Traditional approaches rely on cascaded systems where an Automatic Speech Recognition (ASR) model transcribes audio into text before processing. This pipeline often suffers from error propagation and the loss of nuanced acoustic information. Seal (Speech language model) introduces a novel framework to bridge this gap, transforming frozen LLMs into effective few-shot learners for speech tasks by mapping speech features directly into the LLM's text embedding space.

Architecture and Methodology

The Seal framework utilizes a modular design to leverage the strengths of existing pre-trained models while introducing a specialized alignment strategy.

Model Components

  1. Frozen Speech Encoder: Features are extracted using the encoder from Whisper-large-v2, providing robust acoustic representations.
  2. Trainable Projector: A lightweight linear layer with GELU activation and LayerNorm maps these acoustic features into the word embedding space of the target LLM.
  3. Frozen Language Model: The system was validated using Phi-2 and Phi-3 as the core reasoning engines.

KL-Divergence Alignment and Induction Heads

The primary innovation of Seal lies in its training objective. Rather than simple next-token prediction, the authors employ Kullback-Leibler (KL) divergence loss. This forces the model's output distribution given a speech input ($P(o|s)$) to match the distribution generated by the corresponding text transcript ($P(o|t)$).

To specifically enhance few-shot performance, the training simulates "induction heads"—neural circuits responsible for in-context pattern matching. By feeding the model multiple duplicate transcripts separated by newlines, the projector learns to trigger the same "copy-and-complete" behavior for speech features that the LLM naturally exhibits for text tokens.

Figure 1: The Seal architecture illustrating the KL-divergence loss and the use of duplicate transcripts to simulate induction heads.

Experimental Setup

The model underwent pre-training on approximately 6,600 hours of English speech from Common Voice 14.0 and GigaSpeech XL. To ensure high-quality alignment, Whisper-large-v2 was used to generate "clean" labels, aligning the ground-truth transcripts with the encoder's perception.

Evaluation focused on two primary benchmarks:

  • Fluent Speech Commands (FSC): A dataset for simple intent classification.
  • SLURP: A complex Spoken Language Understanding dataset requiring both intent classification and "slot filling" (entity extraction) across 18 scenarios and 46 actions.

Results and Discussion

Seal demonstrates significant improvements in few-shot speech understanding, particularly when combined with KATE (k-nearest neighbors augmented example selection), which selects acoustically similar in-context examples.

Performance on SLURP

In the challenging SLURP benchmark, Seal-Phi3+KATE achieved notable results in 5-shot scenarios, reaching an intent accuracy of 55.99% and a slot-filling F1 score of 48.09%. This performance highlights the model's ability to extract structured information from raw audio using minimal context.

Figure 3: Performance comparison on the SLURP dataset across 0-shot, 3-shot, and 5-shot settings for intent classification and slot filling.

Key Insights

  • Distribution Matching: Aligning full probability distributions provides a more stable bridge for reasoning than character-level prediction.
  • Mechanistic Alignment: Targeting specific neural circuits like induction heads allows for more efficient transfer of few-shot capabilities from text to speech.
  • Cross-Model Robustness: The method proved effective across different backbone models (Phi-2 and Phi-3), indicating a generalizable approach to multimodal integration.

Conclusion and Future Work

While Seal excels in few-shot settings, a performance gap remains in zero-shot scenarios compared to traditional ASR+LLM pipelines. Future research will focus on scaling the training data—currently a fraction of Whisper's training set—and exploring more diverse prompt structures to further refine the alignment between speech features and the LLM's internal reasoning mechanisms.

Summary updated

BibTeX
@misc{lei2024seal,
  author = {Shuyu Lei and Lingen Liu and Jiaolong Yang and Yasen Jiao and Yuxiang Yang and Yushu Yang and Xiang Guo},
  title = {{Seal: Advancing Speech Language Models to be Few-Shot Learners}},
  year = {2024},
  eprint = {2407.14875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.14875},
}
Figure