Abstract
Knowledge-based authentication is crucial for task-oriented spoken dialogue systems that offer personalised and privacy-focused services. Such systems should be able to enrol (E), verify (V), and identify (I) new and recurring users based on their personal information, e.g. postcode, name, and date of birth. In this work, we formalise the three authentication tasks and their evaluation protocols, and we present EVI, a challenging spoken multilingual dataset with 5,506 dialogues in English, Polish, and French. Our proposed models set the first competitive benchmarks, explore the challenges of multilingual natural language processing of spoken dialogue, and set directions for future research.
Introduction
When interacting with automated task-oriented dialogue systems, such as those used by banks or utility providers, the initial "handshake" of identity verification is a critical hurdle. Traditional systems often rely on rigid menus, which lack the fluidity of human conversation. Furthermore, spoken data is inherently noisy; names are frequently misspelled by Automatic Speech Recognition (ASR), and users provide information in diverse formats (e.g., "the fourth of July" vs. "04/07").
To address these challenges, we formalize the EVI framework: Enrolment (registering a new user), Verification (confirming a claimed identity), and Identification (finding a user in a database).
Methodology
The proposed system architecture consists of ASR, Natural Language Understanding (NLU), Dialogue State Tracking (DST), and a Dialogue Policy. We evaluated two NLU strategies:
- Cautious: Requires a whole-string match, prioritizing precision.
- Seeking: Searches for potentially overlapping substrings, prioritizing recall.
For authentication, we moved beyond binary matching to employ Fuzzy Logic and p-norm fuzzy operators. By calculating scores based on normalized Levenshtein edit distance, the system can handle minor ASR errors gracefully, either proceeding when confident or intelligently re-prompting when data is ambiguous.
The EVI Dataset
We introduce the EVI Dataset, comprising 5,506 dialogues across three locales: British English (en-GB), Polish (pl-PL), and French (fr-FR). Data was collected via crowdsourcing using "fake" profiles. The dataset includes raw audio and n-best transcription lists from Google’s Speech-to-Text, providing a realistic benchmark for navigating ASR errors in multiple languages.
Results and Discussion
A key finding is the impact of entrainment: the system's prompts significantly influence user behavior. For instance, using the NATO phonetic alphabet for postcodes primes users to respond in the same format, facilitating easier ASR processing.
Performance was measured using the Equal Error Rate (EER). The "Seeking" NLU model combined with Fuzzy Verification proved most effective, as it maximized the extraction of correct information from noisy n-best lists.
- Verification: The best model achieved EERs of 11.27% (English), 4.27% (Polish), and 9.11% (French).
- Identification: Using fuzzy retrieval and p-norm operators, we achieved an Identification Rate (IR@1) of 67-95% across languages.
- Efficiency: An "early termination" policy allowed the system to reject impostors 25-30% faster than completing a full dialogue.
Conclusion
While the EVI framework establishes a strong baseline, limitations remain. The scripted nature of the dialogues may not fully capture the behavior of distracted real-world users. Additionally, error propagation between sequential E, V, and I tasks warrants further study. Future work will focus on tighter integration between acoustic models and NLU to better handle out-of-vocabulary entities.
Summary updated