EVI: Multilingual Spoken Dialogue Tasks and Dataset for Knowledge-Based Enrolment, Verification, and Identification

Georgios P. Spithourakis, Ivan Vulic, Michal Lis, Iñigo Casanueva, Pawel Budzianowski

Abstract

Knowledge-based authentication is crucial for task-oriented spoken dialogue systems that offer personalised and privacy-focused services. Such systems should be able to enrol (E), verify (V), and identify (I) new and recurring users based on their personal information, e.g. postcode, name, and date of birth. In this work, we formalise the three authentication tasks and their evaluation protocols, and we present EVI, a challenging spoken multilingual dataset with 5,506 dialogues in English, Polish, and French. Our proposed models set the first competitive benchmarks, explore the challenges of multilingual natural language processing of spoken dialogue, and set directions for future research.

Introduction

In the landscape of automated call centers and voice-based assistants, determining a user's identity is a fundamental prerequisite for providing secure and personalized services. While biometric voiceprints are a frequent subject of academic research, the majority of commercial enterprises continue to rely on knowledge-based authentication (KBA)—verifying identity through personal details such as names, dates of birth, or postcodes.

Despite its ubiquity, KBA in spoken dialogue systems faces significant hurdles due to the inherent "noisiness" of speech. Challenges include Automatic Speech Recognition (ASR) errors, varied formatting (e.g., "first of January" vs. "zero-one-zero-one"), and the linguistic diversity of users. This paper introduces EVI, a comprehensive framework and multilingual dataset designed to formalize and benchmark three core authentication tasks: Enrolment, Verification, and Identification.

The EVI Framework

The researchers formalize the authentication process into three distinct, sequential tasks:

  1. Enrolment (E): The process of registering a new user by collecting and storing their personal information in a database.
  2. Verification (V): A 1:1 matching task where the system confirms a user's claimed identity (e.g., "Are you John Smith?").
  3. Identification (I): A 1:N matching task where the system must determine a user's identity from a database of known profiles without a prior claim (e.g., "Who are you?").

Methodology: NLU and Fuzzy Logic

To address the challenges of spoken input, the study explores two Natural Language Understanding (NLU) strategies:

  • Cautious NLU: This strategy prioritizes precision by requiring a whole-string match. It is most effective during Enrolment, where ensuring the accuracy of the stored data is paramount to prevent "poisoning" the database with incorrect information.
  • Seeking NLU: This strategy prioritizes recall by searching for potentially overlapping substrings. When combined with Fuzzy Logic, it becomes highly effective for Verification and Identification.

To handle the uncertainty of ASR output, the system employs fuzzy matching based on normalized Levenshtein distance (edit distance). By using fuzzy "AND" and "OR" operators, the system aggregates scores across multiple fields (e.g., matching both a name and a postcode) to reach a final authentication decision rather than relying on a binary "yes/no" match.

The EVI Dataset

The EVI dataset is a novel multilingual resource comprising 5,506 dialogues across three locales: British English (en-GB), Polish (pl-PL), and French (fr-FR). The dataset includes:

  • Raw audio recordings and n-best ASR transcriptions.
  • Ground-truth labels for names, postcodes, and dates of birth.
  • 10,000 unique synthetic profiles per locale to ensure privacy and scale.

A key feature of the data collection was the use of priming and entrainment. By varying the bot's prompts, the researchers encouraged users to adopt specific linguistic patterns, such as using the NATO phonetic alphabet for spelling. This approach demonstrated that dialogue design can actively reduce system uncertainty; for instance, over 90% of English speakers adopted the date format suggested by the bot.

Key Results and Insights

The benchmarking results highlight the trade-offs between different NLU strategies and the impact of linguistic structure on authentication performance:

  • Verification Performance: The "Seeking" NLU strategy with fuzzy matching significantly outperformed exact matching. The system achieved an Equal Error Rate (EER) of 11.27% for English. Interestingly, performance was superior in Polish (4.27% EER) and French (9.11% EER), likely because postcodes in these regions are digit-only, reducing ASR complexity compared to alphanumeric British postcodes.
  • Identification Challenges: Identification proved more difficult, with Identification Rates (IR@1) reaching 67% for English and 95% for Polish. The primary bottleneck identified was the "NIL" problem: if the NLU fails to extract a key identifier (like a postcode) correctly on the first attempt, the system often fails to retrieve the correct profile entirely.
  • Operational Efficiency: The study demonstrated that "early termination"—stopping a dialogue once the confidence score falls below a certain threshold—can reduce call duration by 25-30% without compromising security.

Conclusion and Future Work

The EVI framework provides a standardized benchmark for knowledge-based authentication in spoken dialogue. While the current results are promising, the authors identify several areas for future research. These include moving beyond static, scripted policies toward reinforcement learning-based dialogue management and implementing multi-turn belief tracking to allow the system to correct errors made in previous turns. Furthermore, addressing the "NIL" problem in identification remains a critical hurdle for improving the robustness of automated authentication systems.

Summary updated

Abstract

Knowledge-based authentication is crucial for task-oriented spoken dialogue systems that offer personalised and privacy-focused services. Such systems should be able to enrol (E), verify (V), and identify (I) new and recurring users based on their personal information, e.g. postcode, name, and date of birth. In this work, we formalise the three authentication tasks and their evaluation protocols, and we present EVI, a challenging spoken multilingual dataset with 5,506 dialogues in English, Polish, and French. Our proposed models set the first competitive benchmarks, explore the challenges of multilingual natural language processing of spoken dialogue, and set directions for future research.
BibTeX
@inproceedings{spithourakis2022evi,
  author = {Georgios P. Spithourakis and Ivan Vulic and Michal Lis and Iñigo Casanueva and Pawel Budzianowski},
  title = {{EVI: Multilingual Spoken Dialogue Tasks and Dataset for Knowledge-Based Enrolment, Verification, and Identification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=R1W1Gb7JTe},
}
Figure