ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation

Holy Lovenia, Samuel Cahyawijaya, Genta Indra Winata, Peng Xu, Yan Xu, Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham J. Barezi, Pascale Fung

Introduction

Code-switching (CS)—the practice of alternating between two or more languages within a single conversation—is a common feature of bilingual communication. Despite its prevalence, Automatic Speech Recognition (ASR) systems often struggle with CS because they are typically trained on "read speech" recorded in controlled environments. Spontaneous speech, however, is characterized by faster tempos, frequent fillers (e.g., "um," "ah"), and higher spectral variance.

To bridge this gap, the ASCEND (A Spontaneous Chinese-English Dataset) corpus provides 10.62 hours of high-quality, spontaneous bilingual dialogue. This dataset serves as a public benchmark for developing ASR models that can handle the complexities of real-world human interaction.

Dataset Characteristics

The ASCEND corpus was collected from informal, one-on-one conversations among 23 bilingual speakers. To ensure a diverse vocabulary, the researchers employed a session-based approach, prompting participants with topics ranging from technology and philosophy to personal hobbies.

Distribution of speakers by gender across the train, validation, and test splits.

The dataset is partitioned into training, validation, and test sets with disjoint speakers to ensure that models are evaluated on their ability to generalize to unseen voices. As shown in the figure above, the researchers maintained a relatively balanced gender distribution across these splits.

Methodology and Baselines

The authors established baselines using wav2vec 2.0, a self-supervised framework for speech representation learning. They fine-tuned three distinct versions of the model:

  1. Monolingual English: Pre-trained on Common Voice English.
  2. Monolingual Chinese: Pre-trained on Common Voice Chinese.
  3. Multilingual: Pre-trained on a corpus of 53 languages.

Performance was measured using Character Error Rate (CER) for Chinese and Mixed Error Rate (MER), which integrates CER for Chinese and Word Error Rate (WER) for English.

Analysis of Code-Switching

The researchers identified several key linguistic patterns within the ASCEND dataset:

  • Language Turns: A "language turn" occurs when a speaker switches languages within a single utterance. In intra-sentential code-switching instances, speakers averaged approximately 2.18 turns per utterance.
  • L1 Dominance: Despite the bilingual nature of the task, the speakers' first language (Chinese) remained dominant. Chinese segments were notably longer, averaging 8 characters, while English segments averaged 3 words.
  • Topic Influence: Code-switching frequency was highly dependent on the subject matter. For example, "Technology" sessions exhibited the highest rate of intra-sentential switching (31.93%), often due to the use of English technical terminology within Chinese sentences.
An example of inter-sentential code-switching in a multi-turn dialogue.

Results and Discussion

The baseline experiments revealed that the Chinese-pre-trained wav2vec 2.0 model performed best, achieving a 22.69% CER and 27.05% MER on the test set. This suggests that because Chinese constitutes nearly half of the dataset, a monolingual model optimized for the dominant language's phonology can be more effective than a broader multilingual model.

Furthermore, the duration of monolingual utterances for both languages followed a similar distribution, peaking at 1–2 seconds. This consistency reflects the natural rhythm of spontaneous conversational turns.

Limitations and Future Directions

While ASCEND provides a valuable resource for the CS-ASR community, its current scale (10.62 hours) is modest compared to massive monolingual corpora like Librispeech. Future efforts will focus on expanding the dataset's volume and diversity, developing architectures that can better predict language switch points, and applying this data collection methodology to other language pairs.

Summary updated

Abstract

Code-switching is a speech phenomenon occurring when a speaker switches language during a conversation. Despite the spontaneous nature of code-switching in conversational spoken language, most existing works collect code-switching data from read speech instead of spontaneous speech. ASCEND (A Spontaneous Chinese-English Dataset) is a high-quality Mandarin Chinese-English code-switching corpus built on spontaneous multi-turn conversational dialogue sources collected in Hong Kong. We report ASCEND's design and procedure for collecting the speech data, including annotations. ASCEND consists of 10.62 hours of clean speech, collected from 23 bilingual speakers of Chinese and English. Furthermore, we conduct baseline experiments using pre-trained wav2vec 2.0 models, achieving a best performance of 22.69\% character error rate and 27.05% mixed error rate.
BibTeX
@inproceedings{lovenia2022ascend,
  author = {Holy Lovenia and Samuel Cahyawijaya and Genta Indra Winata and Peng Xu and Yan Xu and Zihan Liu and Rita Frieske and Tiezheng Yu and Wenliang Dai and Elham J. Barezi and Pascale Fung},
  title = {{ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation}},
  booktitle = {LREC},
  year = {2022},
  url = {https://openreview.net/forum?id=1Bzczicrghv},
}
Figure