Open-Set Speaker Identification for Unseen Identities

47 concepts34 papers1 proposal

In a traditional closed-set speaker identification system, the model acts like a multiple-choice exam: it is given a voice sample and must choose the most likely candidate from a fixed list of enrolled users. However, real-world environments—such as smart speakers in households or automated call centers—rarely operate under such controlled conditions. These systems face the open-set problem, where they must not only identify known users but also detect and reject "unknown" speakers who were never part of the training or enrollment data.

The Boundary Between Known and Unknown

The fundamental challenge of open-set identification is the lack of a concrete definition for the "unknown" class. While known speakers have specific acoustic signatures, the set of potential imposters is effectively infinite. As Jati et al. observe in Adversarial attack and defense strategies for deep speaker recognition systems, evaluating a system only on closed-set accuracy fails to capture the security risks of real-world biometrics, where distinguishing between enrolled users and unknown imposters is the primary defense.

To bridge this gap, researchers often move away from simple classification toward ranking and rejection mechanisms. For example, Kuroiwa et al. proposed using Earth Mover’s Distance (EMD) to compare voice histograms in Speaker Identification Method Using Earth Mover's Distance for CCC Speaker Recognition Evaluation 2006. By generating a ranked list of matches, the system can reject an input if even the top-ranked candidate fails to meet a specific similarity threshold.

Modeling the Background

If we cannot model every possible unknown speaker, how do we define the "rejection" zone? One common strategy is to use a Universal Background Model (UBM) or a "speaker inventory" to represent the general population.

Angkititrakul et al. refined this in Cluster-dependent modeling and confidence measure processing for in-set/out-of-set speaker identification by clustering known speakers and adapting background models for each cluster. This allows the system to create more precise boundaries around known identities, making it harder for imposters to "mimic" a legitimate user. Similarly, Saeidi et al. argue in their I4u submission to NIST SRE 2012 that modeling unknown non-target speakers remains a critical bottleneck; without a representative "out-of-distribution" set, systems are prone to false acceptances.

Adapting to Dynamic Environments

In many applications, the set of "known" speakers is not static. A robot interacting with people in a room or a meeting transcription service must handle new participants on the fly.

  1. Few-Shot Adaptation: Kishan et al. introduced OpenFEAT in OpenFEAT: Improving Speaker Identification by Open-Set Few-Shot Embedding Adaptation with Transformer. This approach uses a Transformer to transform universal voice embeddings into household-specific ones, allowing the system to distinguish between family members and guests even with very little data.
  2. Sequential Decision Making: In Have we met? MDP based speaker ID for robot dialogue, Krsmanovic et al. treat identification as a sequential process. Instead of making a snap judgment, the system uses a Markov Decision Process to decide if it has enough data to greet a known user or if it should "buy time" to gather more audio from a potentially unknown one.
  3. Joint Recognition and Diarization: Modern end-to-end systems, like the one proposed by Kanda et al. in Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of any Number of Speakers, attempt to count and identify speakers simultaneously. However, the authors note that these systems still struggle when encountering "guest" speakers not present in the initial inventory.

Security in the Open-Set Era

The open-set nature of speaker recognition introduces unique vulnerabilities. Traditional attacks that rely on knowing a victim's identity are less effective when the system can simply reject the input as "unknown."

However, new threats have emerged. Zhao et al. describe a Universal Identity Backdoor Attack in A Universal Identity Backdoor Attack against Speaker Verification based on Siamese Network, where a specific "trigger" voice can bypass security regardless of the enrolled user. Furthermore, Tan et al. found in NRI-FGSM: An Efficient Transferable Adversarial Attack for Speaker Recognition Systems that while adversarial audio can fool closed-set models easily, the complexity of the open-set rejection task creates a "performance gap" that makes black-box attacks significantly harder to execute successfully.

Ultimately, open-set speaker identification works by shifting the focus from classification to differentiation—building a robust understanding of what a specific voice sounds like relative to the vast, unmodeled world of everyone else.

Key Foundational Work

A Speaker Identification Agent (1997) — contributed 2 concept elements to this direction.
A high-performance text-independent speaker identification system based on BCDM (1998) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_open_set_speaker_identification_e8ed55c1,
  title = {open-set speaker identification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/open-set-speaker-identification/rdr_e8ed55c1bf42428c8d434dd297f0be84},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_open_set_speaker_identification_e8ed55c1_v1,
  title = {open-set speaker identification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/open-set-speaker-identification/rdr_e8ed55c1bf42428c8d434dd297f0be84/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake