OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model

Chen Wang, Tianyu Peng, Wenjie Yang, Yun Bai, Guangfu Wang, Jun Lin, Lanpeng Jia, Lingxiang Wu, Jinqiao Wang, Chengqing Zong, Jiajun Zhang2025 · DOI

Abstract

OpenS2S is the first fully open-source, end-to-end Large Speech Language Model (LSLM) specifically designed for empathetic interaction. By combining a streaming interleaved decoding architecture with an automated pipeline for generating high-quality synthetic emotional data, it achieves performance competitive with proprietary models while using significantly fewer training resources. This framework democratizes access to expressive speech-to-speech technology.

Introduction

Human communication is deeply rooted in paralinguistics—the intonation, rhythm, and emotional cues that convey empathy. While Large Language Models (LLMs) have mastered text, most current "Speech-to-Speech" models are either proprietary "black boxes" (like GPT-4o) or open-weights models that focus purely on semantic content, often resulting in flat, robotic responses. Furthermore, training these models typically requires millions of hours of high-quality audio data, creating a massive barrier to entry. OpenS2S aims to bridge this gap by providing a transparent, efficient, and fully open-source framework that understands and generates expressive, empathetic speech.

Architecture and Methodology

The OpenS2S architecture consists of four primary modules integrated into a cohesive end-to-end pipeline:

  1. Audio Encoder: Uses the Qwen2-Audio encoder to transform raw waveforms into meaningful representations at 25Hz, downsampled to 6.25Hz via a speech adapter.
  2. Instruction-Following LLM: The core "brain" is Qwen3-8B-Instruct, which processes interleaved audio and text embeddings.
  3. Streaming Speech Decoder: To achieve low latency, the model uses a framework that interleaves LLM hidden states with generated speech tokens. For every $M$ hidden states consumed, $N$ speech tokens are generated (specifically, a 4:8 ratio). This allows the model to start "speaking" before the full text response is finished.
  4. Token2Wav Decoder: Based on GLM-4-Voice, this module uses a causal flow matching model and a HiFi-GAN vocoder to convert discrete tokens back into high-fidelity audio.
OpenS2S Architecture: Interleaving LLM hidden states with speech tokens for low-latency streaming.

Automated Data Construction

A significant methodological contribution is the Automated Data Construction Pipeline. Instead of relying on expensive human labeling, the authors used a "Self-Instruct" method where an LLM (Qwen3-32B) generates empathetic dialogue scripts. These scripts are then performed by a controllable TTS system (CosyVoice2) that clones voices and applies specific emotional labels (e.g., "elderly," "happy," "sad") to create a diverse, labeled dataset of 100,000 bilingual instruction pairs.

Automated Data Construction Pipeline: Leveraging Self-Instruct and controllable TTS for empathetic data synthesis.

Key Results and Insights

The model was evaluated on VoiceBench (for instruction following) and URO-Bench (for paralinguistic understanding):

  • VoiceBench: OpenS2S achieved a score of 4.09 on the Alpaca subset, outperforming other open models like LLaMA-Omni2 (3.96) and Qwen2-Audio (3.74).
  • URO-Bench: In the "UnderEmotion" subsets, OpenS2S scored 46.90 (EN) and 67.68 (ZH). While slightly behind the massive Kimi-Audio, it significantly outperformed Qwen2-Audio and LLaMA-Omni2, proving its superior grasp of emotional nuances.
  • Efficiency: OpenS2S performs remarkably well despite being trained on a fraction of the data used by commercial competitors, highlighting that high-quality paralinguistic data can be effectively "bootstrapped" using synthetic pipelines.

Conclusion and Future Work

OpenS2S demonstrates that a fully open-source approach can achieve high-quality empathetic speech interaction. While the model currently simulates empathy based on synthetic data, it provides a scalable foundation for future research. Future work will focus on recognizing more subtle emotional shifts and expanding the scale of data synthesis to further close the gap with proprietary models.

Summary updated

BibTeX
@misc{wang2025opens,
  author = {Chen Wang and Tianyu Peng and Wenjie Yang and Yun Bai and Guangfu Wang and Jun Lin and Lanpeng Jia and Lingxiang Wu and Jinqiao Wang and Chengqing Zong and Jiajun Zhang},
  title = {{OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-demos.70},
}
Figure