Target Speech Detection With Multimodal Prompts

Yidi Jiang, Ruijie Tao, Zhengyang Chen, Yanmin Qian, Haizhou LiIEEE Transactions on Audio, Speech and Language Processing 2025 · OpenReview

Abstract

Traditional speaker diarization seeks to detect “who spoke when” according to speaker characteristics. Extending to target speech detection, we detect “when target speech event occurs” according to the semantic characteristics of speech. We propose a novel Multimodal Target Speech Detection (MM-TSD) framework, which accommodates diverse and multimodal prompts to specify target speech events in a flexible and user-friendly manner, including semantic language description, pre-enrolled speech, pre-registered face image, and audio-language logical prompts. We further propose a voice-face aligner module to project human voice and face representation into a shared space. We develop a multimodal dataset based on VoxCeleb2 for MM-TSD training and evaluation. Additionally, we conduct comparative analysis and ablation studies for each category of prompts to validate the efficacy of each component in the proposed framework. Furthermore, our framework demonstrates versatility in performing various signal processing tasks, including speaker diarization and overlap speech detection, using task-specific prompts. MM-TSD achieves robust and comparable performance as a unified system compared to specialized models. Moreover, MM-TSD shows capability to handle complex conversations for real-world dataset.
BibTeX
@article{jiang2025target,
  author = {Yidi Jiang and Ruijie Tao and Zhengyang Chen and Yanmin Qian and Haizhou Li},
  title = {{Target Speech Detection With Multimodal Prompts}},
  journal = {IEEE Transactions on Audio, Speech and Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=RyiwUqPqEs},
}

Loading related content...

Figure