VN-SLU: A Vietnamese Spoken Language Understanding Dataset

Tuyen Tran, Khanh Le, Ngoc Dang Nguyen, Minh Nhat Vu, Huyen Ngo, Woomyoung Park, Thi Thu Trang Nguyen2024 · DOI

TL;DR

This paper introduces VN-SLU, the first large-scale dataset for Vietnamese Spoken Language Understanding (SLU), addressing the scarcity of resources for low-resource languages. They developed a novel crowdsourcing pipeline for data collection and demonstrated that incorporating pitch information significantly improves SLU performance for Vietnamese.

What Problem Does This Paper Address?

Spoken Language Understanding (SLU) is crucial for enabling machines to comprehend and act on spoken commands. While English has a wealth of SLU datasets, low-resource languages like Vietnamese are severely underserved. This lack of data hinders the development and evaluation of SLU systems for these languages. The authors aim to bridge this gap by creating a comprehensive Vietnamese SLU dataset and exploring methods to improve SLU performance for this language.

Methods

The authors developed a multi-stage pipeline for constructing the VN-SLU dataset. This involved:

  • Data Preparation: Identifying 28 distinct intents and 57 entity values relevant to virtual assistant and smart home scenarios.
  • Scenario Generation: A novel algorithm (Algorithm 1) was used to generate diverse and natural-sounding command combinations. This algorithm focuses on mapping natural slot value associations (e.g., "close" with "door") and then combining these into realistic utterances, ensuring a balanced distribution of intents and slots. They also incorporated system number entities (like "target number") and temporal slots ("time at") to enhance realism.
  • Audio Recording: A crowdsourcing web tool was developed, allowing participants to record speech utterances for generated scenarios. This tool included a recording feature and a validation mechanism to ensure data quality and alignment between spoken words and transcribed intents/slots.
  • Data Post-processing: This stage likely involved cleaning and organizing the collected data.

A key methodological contribution is the integration of pitch information into SLU models. Recognizing that pitch is crucial for meaning in tonal languages like Vietnamese, they proposed a model architecture that concatenates features from a standard audio encoder (HuBERT) with features extracted from a dedicated pitch encoder. This pitch encoder uses normalized cross-correlation and median smoothing, followed by CNN and RNN stacks.

Datasets

The paper introduces the VN-SLU dataset, which comprises:

  • 17,321 utterances
  • From 240 Vietnamese speakers
  • Totaling 20.07 hours of audio
  • Covering SmartHome and VirtualAssistant domains.

The dataset is publicly available. The authors also compare VN-SLU to existing SLU datasets like SLURP, Snips, and FSC, highlighting its unique contribution to low-resource languages.

Key Insights and Ideas

  • The critical need for low-resource language datasets: The paper strongly emphasizes the disparity in available resources for languages like Vietnamese compared to English, highlighting VN-SLU's importance for democratizing SLU research.
  • Crowdsourcing for quality and diversity: The novel crowdsourcing pipeline, including the scenario generation algorithm and validation tool, was crucial for obtaining high-quality, diverse, and naturalistic data.
  • Pitch is paramount for Vietnamese SLU: The finding that integrating pitch information significantly boosts SLU performance underscores the importance of linguistic-specific features for tonal languages. This suggests that models trained solely on acoustic or textual features might miss crucial semantic nuances.
  • Challenging test set sampling: The authors experimented with different test set sampling strategies and found that a "Speaker-Value" approach (where speakers in the test set are entirely new and have unique values) creates the most challenging evaluation, effectively highlighting model limitations.

Key Results

  • The proposed VN-SLU dataset contains 17,321 utterances from 240 speakers, totaling 20.07 hours of audio.
  • When evaluating different sampling strategies for test sets, the "Speaker-Value" method proved most challenging, leading to lower accuracy scores for SLU models, thus serving as a better benchmark for model robustness.
  • Integrating pitch information into a Speechbrain-HuBERT model resulted in notable improvements:
  • Intent Accuracy increased by 1.67%.
  • SLU-F1 score increased by 2.19%.
  • The NeMo Conformer Transformer model achieved the highest intent accuracy (91.81%) and SLU-F1 (67.82%) on the VN-SLU dataset, though utterance accuracy remained lower (37.57%).

Limitations and Future Work

  • Utterance Accuracy: While intent and SLU-F1 scores were promising, the utterance accuracy remained relatively low, indicating room for improvement in end-to-end prediction.
  • Domain Specificity: The current dataset focuses on SmartHome and VirtualAssistant domains. Future work could expand to other domains to create a more general-purpose Vietnamese SLU dataset.
  • Model Architectures: The authors explored RNN-based and Conformer-based models. Further research could investigate other advanced architectures and self-supervised learning techniques tailored for Vietnamese.
  • Pitch Integration: While pitch integration showed promise, exploring more sophisticated methods for pitch feature extraction and integration could yield further gains.

Summary updated

BibTeX
@misc{tran2024vn,
  author = {Tuyen Tran and Khanh Le and Ngoc Dang Nguyen and Minh Nhat Vu and Huyen Ngo and Woomyoung Park and Thi Thu Trang Nguyen},
  title = {{VN-SLU: A Vietnamese Spoken Language Understanding Dataset}},
  year = {2024},
  doi = {10.21437/interspeech.2024-1976},
}
Figure