An Exploration of In-Context Learning for Speech Language Model

Ming-Hao Hsu, Kai-Wei Chang, Shang-Wen Li, Hung-yi Lee

Introduction

In Natural Language Processing (NLP), Large Language Models (LLMs) like GPT-4 have demonstrated a remarkable "emergent ability" known as In-Context Learning (ICL). This allows models to perform new tasks by simply observing a few examples in a prompt, without any parameter updates. However, this capability has remained largely absent in speech processing. Most existing speech systems are task-specific, requiring extensive fine-tuning for each new application, such as switching from command recognition to emotion detection.

The difficulty in bringing ICL to speech stems from two primary factors: the lack of a natural discrete vocabulary in speech (the "Modality Gap") and the relatively small scale of current speech language models (LMs) compared to their trillion-parameter text counterparts. This paper introduces the first successful application of ICL to "textless" speech language models, enabling them to learn new classification tasks on the fly.

Methodology: Enabling ICL in Speech

The researchers utilize a Generative Spoken Language Model (GSLM) as their foundation. This textless model operates by converting raw audio into discrete speech tokens—essentially "audio words" identified through self-supervised learning—and predicting subsequent tokens. To bridge the gap to ICL, a two-stage approach is proposed.

Random Label Mapping (RLM) and Warmup Training

Because the model lacks textual understanding, task labels (e.g., "Happy", "English") are mapped to specific discrete speech tokens via Random Label Mapping (RLM). Crucially, the authors found that standard speech LMs do not possess ICL capabilities "out of the box." To address this, they implement Warmup Training.

During this phase, a tiny fraction of the model's parameters (less than 0.1%) are trained using prompt tuning on a variety of speech tasks. This process does not teach the model specific tasks but rather the meta-skill of utilizing demonstrations $[x_1, y_1, ... x_n, y_n]$ to predict the label of a new target $x_t$. Once this warmup is complete, the model is frozen for in-context inference.

Overview of the Warmup Training and In-context Learning framework.

Experimental Setup

The model's performance was evaluated across seven diverse speech classification datasets, including Speech Command Recognition (SCR), Emotion Recognition (ER), Language Identification (LID), and Sarcasm Detection (SD). To test cross-task generalization, these datasets were divided into two groups. The model underwent warmup training on one group and was then tested on the other, entirely unseen group.

Key Results and Analysis

The introduction of warmup training led to a dramatic improvement in ICL performance. For instance, on the Arabic SC dataset, accuracy surged from a baseline of 3.9% (without warmup) to 40.9%, significantly exceeding the random guessing rate of 28.0%.

Performance comparison across various tasks with and without warmup training.

Generalization and Robustness

The model demonstrated a strong ability to generalize to unseen tasks. After warming up on command recognition and language ID, it achieved 47.7% accuracy on Emotion Recognition (IEMOCAP), a task it had never encountered during training. Furthermore, the model's "guessing rate"—the frequency with which it selects a label present in the provided demonstrations—exceeded 90%, confirming that it truly understands the ICL format.

Insights into the "Analogy" Mechanism

Analysis of attention weights revealed that the model actively "reads" demonstrations. It processes audio examples and the target audio before focusing on the labels within the demonstrations to formulate a prediction. The researchers also noted that standardization is vital; fixing audio segments to a length of 50 tokens provided the optimal balance of information and sequence manageability.

Conclusion and Future Work

This work proves that ICL is achievable for speech LMs, though it currently requires a warmup phase due to the smaller scale of existing models. While the results are competitive with classical baselines like Support Vector Classifiers (SVC), challenges remain regarding sequence length and cross-lingual generalization. Future research will explore scaling these models further and extending ICL to generative speech tasks such as translation and summarization.

Summary updated

BibTeX
@inproceedings{hsu2024exploration,
  author = {Ming-Hao Hsu and Kai-Wei Chang and Shang-Wen Li and Hung-yi Lee},
  title = {{An Exploration of In-Context Learning for Speech Language Model}},
  booktitle = {Interspeech},
  year = {2024},
  doi = {10.21437/interspeech.2024-1932},
}
Figure