Contextual Face Recognition with a Nested-Hierarchical Nonparametric Identity Model

Daniel Coelho de Castro, Sebastian NowozinarXiv 2018 · OpenReview

Abstract

Current face recognition systems typically operate via classification into known identities obtained from supervised identity annotations. There are two problems with this paradigm: (1) current systems are unable to benefit from often abundant unlabelled data; and (2) they equate successful recognition with labelling a given input image. Humans, on the other hand, regularly perform identification of individuals completely unsupervised, recognising the identity of someone they have seen before even without being able to name that individual. How can we go beyond the current classification paradigm towards a more human understanding of identities? In previous work, we proposed an integrated Bayesian model that coherently reasons about the observed images, identities, partial knowledge about names, and the situational context of each observation. Here, we propose extensions of the contextual component of this model, enabling unsupervised discovery of an unbounded number of contexts for improved face recognition.

Introduction

Most modern face recognition systems treat identification as a standard closed-set classification problem. Given a fixed list of names, the system attempts to assign a label to a detected face. However, this approach diverges significantly from human cognitive processes. Humans can recognize a "familiar face" (identity) even without knowing the person's name (tagging). Crucially, humans leverage situational context—such as expecting to see family at home or colleagues at work—to disambiguate similar-looking individuals.

Conceptual overview of identities shared across multiple situational contexts like Home, Work, and Gym.

Standard systems often fail to utilize unlabeled data and lack the mechanism to incorporate these situational "priors." This research proposes a shift toward an open-world model that learns about people and places dynamically, discovering an unbounded number of identities and contexts entirely unsupervised.

Methods

The proposed architecture is a nested-hierarchical nonparametric Bayesian model composed of four primary layers:

  1. Identity Model: Utilizing a Dirichlet Process (DP), the model handles an unbounded population. It does not require a predefined number of individuals; instead, it creates new identity clusters when an observation does not fit existing ones.
  2. Context Model: This is the core extension of the current work. By employing a Nested-Hierarchical Dirichlet Process (n-HDP), the system can discover an unbounded number of situational contexts (e.g., a new hobby group or vacation spot) without manual intervention.
  3. Face Model: The system uses compact face embeddings generated by a pre-trained neural network (OpenFace) rather than raw pixels. These embeddings are modeled using Gaussian distributions.
  4. Label Model: This semi-supervised component manages sparse and potentially noisy name tags. It learns to propagate names to all observations within the same identity cluster.
Probabilistic graphical model of the nested-hierarchical nonparametric identity framework.

Key Insights

  • Decoupling Recognition and Tagging: Matching a face to a previously seen cluster (recognition) is treated as an unsupervised task, while attaching a name (tagging) is a separate, semi-supervised process.
  • Context as a Grouping Mechanism: Contexts are defined as clusters of frames that share similar identity distributions. If certain individuals frequently appear together, the model learns they belong to the same context.
  • Nonparametric Flexibility: By using nonparametric priors (GEM and DP distributions), the model avoids the "Model Selection" problem, automatically determining the appropriate number of people and locations from the data.
  • Integration of Environmental Cues: The Bayesian framework allows for the direct inclusion of non-visual data, such as GPS coordinates and timestamps, to aid in context discovery.

Key Results

This work provides a mathematical formulation for a generative process that enables global identity sharing across contexts while maintaining context-specific weights. The model can partition a photo collection into an arbitrary number of contexts based on co-occurrence patterns and metadata. This allows for improved recognition accuracy by leveraging situational priors, even in the absence of explicit labels.

Limitations and Future Work

While theoretically robust, Bayesian nonparametric models can be computationally intensive, requiring advanced inference techniques like Markov Chain Monte Carlo or Variational Inference. Furthermore, the system's performance remains dependent on the quality of the underlying face embeddings; if the embedding space cannot distinguish between two individuals, the Bayesian layer will struggle to separate them. Future work will focus on large-scale empirical validation of incorporating environmental cues like GPS and time.

Summary updated

Abstract

Current face recognition systems typically operate via classification into known identities obtained from supervised identity annotations. There are two problems with this paradigm: (1) current systems are unable to benefit from often abundant unlabelled data; and (2) they equate successful recognition with labelling a given input image. Humans, on the other hand, regularly perform identification of individuals completely unsupervised, recognising the identity of someone they have seen before even without being able to name that individual. How can we go beyond the current classification paradigm towards a more human understanding of identities? In previous work, we proposed an integrated Bayesian model that coherently reasons about the observed images, identities, partial knowledge about names, and the situational context of each observation. Here, we propose extensions of the contextual component of this model, enabling unsupervised discovery of an unbounded number of contexts for improved face recognition.
BibTeX
@misc{castro2018contextual,
  author = {Daniel Coelho de Castro and Sebastian Nowozin},
  title = {{Contextual Face Recognition with a Nested-Hierarchical Nonparametric Identity Model}},
  year = {2018},
  eprint = {1811.07753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.07753},
}
Figure