MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling

Yifan Cheng, Ruoyi Zhang, Jiatong Shi

Introduction

Standard speech models are typically trained on millions of hours of data, yet emotional speech datasets remain a significant bottleneck in the field. Existing datasets like IEMOCAP or MELD are relatively small, often totaling less than 300 hours, primarily because they rely on manual human annotation. Human labeling is inherently slow, expensive (costing roughly $10.00 per hour), and prone to inconsistency, with low inter-annotator agreement. Furthermore, most datasets are limited to "Ekman’s basic emotions" (e.g., Joy, Anger, Fear), which lacks the granularity required for high-quality, expressive text-to-speech (TTS) systems.

To address these challenges, we introduce MIKU-PAL (Multimodal Intelligence Kit for Understanding - Paralinguistic and Affect Labeling), a fully automated multimodal pipeline. MIKU-PAL leverages Multimodal Large Language Models (MLLMs) to label emotional speech with human-level accuracy and "near-perfect" consistency, scaling emotion categories to a nuanced set of 26.

The MIKU-PAL Pipeline

MIKU-PAL automates emotion labeling through a modular three-stage process designed to ensure high-fidelity data extraction from raw video inputs.

The MIKU-PAL automated multimodal pipeline architecture.
  1. Audio Preprocessing: The system utilizes MDX-Net for music source separation to remove background noise and music, ensuring the model receives "clean speech." It then uses Whisper-large v3 for precise transcription and temporal alignment.
  2. Vision Preprocessing: To ensure the analyzed emotion belongs to the active speaker, the pipeline employs face detection ($S^3$FD or DSFD) and TalkNet for active speaker identification. This step is crucial; ablation studies showed that visual preprocessing improves annotation accuracy by 25.6% on the MELD dataset.
  3. MLLM Emotion Analysis: The core reasoning is performed by Gemini 2.0 Flash. Using a structured system prompt, the model selects from 26 emotion categories and provides a textual rationale for its choice based on facial cues, body language, and voice cues.

MIKU-EmoBench: A New Frontier in Emotional Data

Using this pipeline, we produced MIKU-EmoBench, a newly released dataset containing 131.2 hours of emotional speech across 65,970 utterances sourced from YouTube. Unlike traditional datasets, MIKU-EmoBench covers 26 fine-grained emotion categories, such as Adoration, Nostalgia, Craving, and Entrancement.

Visualization of the 26 fine-grained emotion categories in the MIKU-EmoBench dataset.

This increased granularity allows for significantly more expressive TTS. Fine-tuning the Fish-Speech model on MIKU-EmoBench resulted in a boost in Mean Opinion Score (MOS) and a higher Emotion Similarity score (0.92 compared to 0.88 for the baseline).

Key Results and Insights

The MIKU-PAL system demonstrates several advantages over traditional human labeling:

  • Standardized Consistency: MIKU-PAL achieved a Fleiss $\kappa$ score of 0.93–0.95, indicating "near-perfect" agreement. In contrast, human annotators typically score between 0.40 and 0.43. This suggests that MLLMs can act as a highly standardized "ruler" for emotion.
  • Human-Level Accuracy: The system achieved 68.5% accuracy on the MELD dataset. When ambiguous legacy labels (like "neutral" vs "frustration") are excluded, accuracy jumps to approximately 75%, surpassing average human performance. Human raters validated 83% of the automated annotations as "rational."
  • Economic and Engineering Efficiency: Labeling an hour of data costs less than $0.50 with MIKU-PAL. On a workstation with 8x RTX 4090s, the system processes video at a 1:12 ratio, meaning one hour of video is processed in just five minutes.

Limitations and Future Work

While MIKU-PAL represents a significant step forward, it is currently dependent on the underlying MLLM (Gemini 2.0 Flash), which may introduce its own biases or hallucinations. Additionally, sourcing data from YouTube may inherit platform-specific demographic or content biases. Vision preprocessing remains the primary computational bottleneck. Future work will focus on enhancing the pipeline's robustness for even more diverse "in-the-wild" video conditions and reducing the computational overhead of face tracking.

Summary updated

BibTeX
@inproceedings{cheng2025miku,
  author = {Yifan Cheng and Ruoyi Zhang and Jiatong Shi},
  title = {{MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling}},
  booktitle = {Interspeech},
  year = {2025},
  doi = {10.21437/interspeech.2025-648},
}
Figure