Introduction
Live-streaming platforms face significant challenges in monitoring audio streams for "abnormal events," such as policy violations or inappropriate language. Traditional Spoken Language Understanding (SLU) systems rely on a two-stage pipeline: an Automatic Speech Recognition (ASR) model to transcribe speech to text, followed by a Natural Language Processing (NLP) model to classify intent. However, this approach is computationally expensive and prone to "error cascading," where transcription mistakes degrade downstream classification. Furthermore, text transcripts lose vital acoustic cues like prosody and tone.
The Ensemble End-to-End Approach
To overcome these limitations, this research proposes an Ensemble End-to-End (E2E) SLU system. Unlike traditional models, this architecture maps raw audio directly to intent without intermediate text transcription.
The system employs two distinct pre-trained ASR models—a Hybrid CTC/attention model and a Transformer-based model—to extract high-dimensional feature vectors. By concatenating these internal representations, the model captures a richer set of phonetic and semantic information than any single model could provide. These features are then processed by a two-layer Bi-directional LSTM network.
A critical innovation in this architecture is the replacement of the standard attention layer with a max-pooling layer. While attention is effective for short clips, it struggles with the 15-second audio segments used in this study, which contain a high volume of time steps. Max-pooling proves more robust at isolating sparse "abnormal" signals within these long, noisy sequences.
Dataset: Kuaishou Abnormal Event Dataset
The study introduces the Kuaishou Abnormal Event Dataset, a large-scale collection of 52,446 real-world audio clips from the Kuaishou live-streaming platform. Each clip is 15 seconds long and contains unscripted speech, heavy background noise, and various regional accents. Notably, the dataset contains no text transcripts, necessitating an E2E approach that bypasses the transcription bottleneck.
Results and Performance
The ensemble approach significantly outperforms baseline E2E models. The transition from a standard attention mechanism to max-pooling alone improved the F1 score from 68.00% to 75.14%.
As shown in the results, the final ensemble model achieved an F1 score of 77.73%, representing a 9.7% performance boost over previous single-model baselines. The high precision (89.52%) indicates the model is particularly effective at minimizing false positives, which is crucial for automated moderation in a live-streaming environment.
Conclusion
By leveraging ensemble feature extraction and optimized pooling layers, this system provides a scalable solution for real-time abnormal event detection in Chinese audio streams. The findings highlight that for long-form, noisy audio, architectural simplicity (like max-pooling) and feature diversity (via ensembling) can outperform traditional attention-based NLP paradigms.
Summary updated