TL;DR
This paper tackles the challenge of task conflicts in multi-task learning (MTL) for stuttering detection. They introduce a novel rule-based MTL strategy and a Multi-Mixture-of-Experts (MMoE) approach to effectively manage these conflicts, leading to significant improvements in stuttering detection performance and outperforming state-of-the-art methods.
What Problem Does This Paper Address?
Stuttering is a speech disorder affecting millions, and automated detection holds significant value for improving communication and mental health. Multi-Task Learning (MTL) has shown promise in stuttering detection by leveraging the correlations between different stuttering symptoms (like prolongations, blocks, repetitions, and interjections). However, a key challenge in MTL is "task conflict," where the learning signals from different tasks can interfere with each other, hindering overall performance. This paper specifically addresses this issue by identifying and mitigating these conflicts in stuttering detection, aiming to improve the accuracy and robustness of automated systems.
Methods
The authors propose two main strategies to address task conflicts in stuttering detection:
- Rule-based MTL Strategy: This approach analyzes the inherent relationships between stuttering tasks. Based on this analysis, they group tasks that are more highly correlated into a single MTL group (specifically, /p, /wr, and /i tasks) while treating less correlated tasks (/b and /r) as independent. This aims to reduce interference by treating tasks with similar characteristics together.
- MMoE-based MTL Strategy: This method utilizes a Multi-Mixture-of-Experts (MMoE) framework. Instead of a single shared network for all tasks, MMoE employs multiple "expert" networks, each specializing in different aspects of the data. Gating networks then learn to dynamically route information from these experts to each specific task. This allows for more flexible learning, enabling the model to leverage shared information while also accommodating task-specific nuances and reducing conflicts.
Both strategies build upon a baseline 12-layer Conformer encoder with a single-layer BiLSTM head.
Datasets
The primary dataset used in this study is the AS-70 dataset. This dataset consists of recordings from 70 native Mandarin-speaking adults who stutter (AWS). The recordings include both dialogues and reading speech commands. The dataset is annotated with five types of stuttering events: /wr (word/phrase repetition), /b (block), /p (prolongation), /r (sound repetition), and /i (interjection). The paper also mentions using the dataset split method from the SLT2024 competition.
Key Insights and Ideas
- Task Conflicts are Real and Detrimental: The study provides strong evidence that task conflicts exist in stuttering detection MTL. Specifically, the /b (block) and /r (sound repetition) tasks show significant gradient conflicts with other tasks, negatively impacting performance when treated within a standard MTL framework.
- Correlation vs. Conflict: While correlations between stuttering tasks are beneficial for MTL, ignoring potential conflicts can lead to performance degradation. The paper highlights that simply grouping all tasks together in an MTL setup can sometimes hurt performance for certain tasks (e.g., /b and /r).
- Targeted Grouping Works: The rule-based MTL strategy, by grouping more correlated tasks (/p, /wr, /i) and treating others independently, demonstrates that a more nuanced approach to task grouping can be effective.
- MMoE Offers Flexible Specialization: The MMoE framework proves adept at mitigating task conflicts by allowing experts to specialize and gating networks to intelligently route information. This leads to significant improvements, particularly for tasks that were previously hindered by conflicts.
Key Results
- Rule-based MTL Improvement: The rule-based MTL strategy improved the average F1 score by 19.9% over the baseline on the AS-70 dataset.
- MMoE-based MTL Further Enhancement: The MMoE-based MTL strategy further boosted performance, improving the average F1 score by an additional 7.55% over the rule-based approach.
- State-of-the-Art Performance: The proposed methods, particularly the MMoE-MTL, achieved state-of-the-art results, outperforming existing approaches and showing an 11.27% improvement in the overall F1 score compared to the best previous method.
- Task-Specific Gains: The MMoE-MTL strategy showed particularly notable gains in the /b task, which was previously a bottleneck.
Limitations and Future Work
- Limitations: The paper acknowledges that while MMoE mitigates conflicts, there might still be deeper relationships between tasks that could be further exploited. The current analysis of task conflicts is primarily based on gradient cosine similarity.
- Future Work: The authors plan to investigate deeper relationships between tasks to improve interpretability. They also aim to further optimize performance by integrating acoustic signals with semantic information and exploring the model's application and scalability on other stuttering datasets.
Summary updated