Ensemble Chinese End-to-End Spoken Language Understanding for Abnormal Event Detection from audio stream

Haoran Wei, Fei Tao, Runze Su, Sen Yang, Ji Liu

Introduction

Live-streaming platforms face significant challenges in monitoring audio streams for "abnormal events," such as policy violations or inappropriate language. Traditional Spoken Language Understanding (SLU) systems rely on a two-stage pipeline: an Automatic Speech Recognition (ASR) model to transcribe speech to text, followed by a Natural Language Processing (NLP) model to classify intent. However, this approach is computationally expensive and prone to "error cascading," where transcription mistakes degrade downstream classification. Furthermore, text transcripts lose vital acoustic cues like prosody and tone.

The Ensemble End-to-End Approach

To overcome these limitations, this research proposes an Ensemble End-to-End (E2E) SLU system. Unlike traditional models, this architecture maps raw audio directly to intent without intermediate text transcription.

The proposed Ensemble End-to-End SLU architecture utilizes dual ASR models for robust feature extraction.

The system employs two distinct pre-trained ASR models—a Hybrid CTC/attention model and a Transformer-based model—to extract high-dimensional feature vectors. By concatenating these internal representations, the model captures a richer set of phonetic and semantic information than any single model could provide. These features are then processed by a two-layer Bi-directional LSTM network.

A critical innovation in this architecture is the replacement of the standard attention layer with a max-pooling layer. While attention is effective for short clips, it struggles with the 15-second audio segments used in this study, which contain a high volume of time steps. Max-pooling proves more robust at isolating sparse "abnormal" signals within these long, noisy sequences.

Dataset: Kuaishou Abnormal Event Dataset

The study introduces the Kuaishou Abnormal Event Dataset, a large-scale collection of 52,446 real-world audio clips from the Kuaishou live-streaming platform. Each clip is 15 seconds long and contains unscripted speech, heavy background noise, and various regional accents. Notably, the dataset contains no text transcripts, necessitating an E2E approach that bypasses the transcription bottleneck.

Results and Performance

The ensemble approach significantly outperforms baseline E2E models. The transition from a standard attention mechanism to max-pooling alone improved the F1 score from 68.00% to 75.14%.

Performance comparison between the single-model baseline and the proposed ensemble E2E SLU system.

As shown in the results, the final ensemble model achieved an F1 score of 77.73%, representing a 9.7% performance boost over previous single-model baselines. The high precision (89.52%) indicates the model is particularly effective at minimizing false positives, which is crucial for automated moderation in a live-streaming environment.

Conclusion

By leveraging ensemble feature extraction and optimized pooling layers, this system provides a scalable solution for real-time abnormal event detection in Chinese audio streams. The findings highlight that for long-form, noisy audio, architectural simplicity (like max-pooling) and feature diversity (via ensembling) can outperform traditional attention-based NLP paradigms.

Summary updated

BibTeX
@inproceedings{wei2021ensemble,
  author = {Haoran Wei and Fei Tao and Runze Su and Sen Yang and Ji Liu},
  title = {{Ensemble Chinese End-to-End Spoken Language Understanding for Abnormal Event Detection from audio stream}},
  booktitle = {ICBDT},
  year = {2021},
  doi = {10.1145/3490322.3490323},
}
Figure