Introduction
Speaker Verification (SV) is a cornerstone of biometric security, widely used in applications ranging from financial services to smart home assistants. Traditionally, SV is handled by specialized Deep Neural Networks, such as ECAPA-TDNN, which extract unique "voiceprints" (embeddings) to compare audio segments.
With the rise of Audio Large Language Models (ALLMs) like Qwen2-Audio, which excel at speech recognition and audio captioning, a critical question arises: can these models—primarily trained to understand what is being said—effectively distinguish who is saying it? This research explores the gap in ALLMs' "identity-blindness" in zero-shot settings and proposes methods to bridge it.
Methodology
The researchers reformulated Speaker Verification as an Audio Question-Answering (QA) task. Instead of calculating mathematical similarity scores, the model is prompted: "Please determine whether the above two audio segments are from the same speaker."
Prompting and Fine-Tuning
Technical contributions focused on optimizing how audio is presented to the model. Four strategies were evaluated: "Separate" (distinct inputs), "Concat" (joined), "Concat + Silence" (joined with a 1-second gap), and "Mix" (overlaid). The Concat + Silence strategy proved most effective, likely because the silence acts as a clear delimiter for the model's attention mechanism.
To improve performance without retraining the entire model, the authors employed Low-Rank Adaptation (LoRA) to fine-tune only 18 million parameters. A key innovation was Rule-based Hard Pair Sampling, where the model was trained on "hard" examples—different speakers who share the same gender, age, or dialect—forcing the model to learn subtle idiosyncratic features rather than relying on superficial cues like pitch.
Experimental Results
Evaluation across major benchmarks like VoxCeleb, CN-Celeb, and 3D-Speaker revealed that while zero-shot performance is often near chance (50-60%), targeted fine-tuning yields dramatic improvements.
Key Findings
- Fine-Tuning Efficacy: On the VoxCeleb dataset, fine-tuning improved Kimi-Audio’s accuracy from approximately 70% to 95% for gender-matched pairs.
- Superiority in Short Durations: One of the most significant findings was that fine-tuned ALLMs outperformed the industry-standard ECAPA-TDNN on very short audio clips (under 2 seconds). The ALLM achieved 80.90% accuracy compared to the baseline's 78.80%, suggesting that the broad pre-trained context of ALLMs provides an advantage when acoustic data is sparse.
- Hard Sampling Impact: Utilizing the hard pair strategy instead of random sampling led to performance gains of 2-8% across challenging dimensions like dialect and recording distance.
- Unified Multi-Tasking: Unlike traditional cascaded systems, a single fine-tuned ALLM can verify speaker identity and transcribe content simultaneously, achieving 98.87% overall accuracy on combined text-dependent SV tasks.
Conclusion and Future Work
While a performance gap still exists between ALLMs and specialized models in long-duration or "clean" scenarios, this study demonstrates that ALLMs can be highly effective SV systems with lightweight fine-tuning. Future research will focus on reducing the computational footprint of these 8-billion parameter models to enable real-time deployment on resource-constrained edge devices.
Summary updated