Uncertainty Modeling of Contextual-Connection between Tracklets for Unconstrained Video-based Face Recognition

Jingxiao Zheng, Ruichi Yu, Jun-Cheng Chen, Boyu Lu, Carlos Domingo Castillo, Rama Chellappa

TL;DR

This paper introduces the Uncertainty-Gated Graph (UGG), a novel graphical model for unconstrained video-based face recognition. UGG tackles the challenge of noisy contextual information by adaptively modeling uncertainty in connections between face tracklets, leading to state-of-the-art performance on challenging datasets.

What Problem Does This Paper Address?

Recognizing faces in videos is significantly harder than in still images due to variations in pose, blur, and occlusion. A common strategy is to leverage contextual information, like body appearance, to propagate identity information from clear faces to blurry or occluded ones within a video. However, previous methods often suffer from propagating incorrect information because they treat these contextual connections as fixed and reliable, even when they are noisy or misleading. This paper aims to address this by explicitly modeling the uncertainty in these contextual connections.

Methods

The core contribution is the Uncertainty-Gated Graph (UGG). UGG represents each detected and tracked face sequence (tracklet) as a node in a graph. Connections between these tracklets are modeled as edges. What's novel is the introduction of uncertainty gates on these edges. These gates are not fixed; they are adaptively updated during inference based on the identity distributions of the connected tracklets.

Specifically, UGG uses two types of gates:

  • Positive gates: These encourage tracklets with similar contextual information (e.g., body appearance) to be assigned the same identity. Their confidence is influenced by the similarity of the tracklets' identity distributions.
  • Negative gates: These discourage tracklets that are clearly different (e.g., from the same frame but different people) from being assigned the same identity. These are based on "cannot-link" constraints derived from detection information and are fixed during inference.

The model uses a graphical model framework with an energy function that balances unary potentials (prior identity information from still-image face recognition) and pairwise potentials (interactions between tracklets mediated by the adaptive gates). Inference is performed using a message-passing approach (mean-field approximation) to iteratively update both tracklet identity distributions and gate probabilities. UGG can be used as a post-processing step at inference time or integrated into an end-to-end deep learning architecture for supervised and semi-supervised training.

Datasets

The paper evaluates its method on two challenging datasets:

  • Cast Search in Movies (CSM): A large-scale dataset for person search in movies, used for evaluating tracklet identification.
  • IARPA Janus Surveillance Video Benchmark (IJB-S): An unconstrained video face recognition dataset known for its challenging low-quality surveillance videos.

Key Insights and Ideas

  • Adaptive connections are crucial: The paper strongly demonstrates that simply using fixed contextual connections (like body similarity) is insufficient and can lead to errors. The ability to adaptively adjust the strength of these connections based on evolving identity information is key.
  • Modeling uncertainty pays off: Explicitly modeling the uncertainty in contextual connections, rather than treating them as absolute, significantly improves robustness and accuracy in unconstrained scenarios.
  • Negative connections are powerful: The inclusion of negative gates, which actively discourage incorrect identity assignments, provides a substantial performance boost, highlighting their importance in disambiguating identities.
  • Flexibility of UGG: The model's ability to be used either as a standalone inference-time module or as a trainable component within a larger deep learning system makes it highly versatile.

Key Results

  • State-of-the-art performance: UGG achieves state-of-the-art results on both the CSM and IJB-S datasets across various evaluation metrics (mAP, R@k, EERR).
  • Significant improvements over baselines: UGG consistently outperforms simpler methods like FACE (using only facial similarity) and PPCC (a previous graph-based method) by substantial margins. For example, on CSM, UGG-T(favg) shows significant improvements over PPCC. On IJB-S, UGG-U(favg) outperforms PPCC by a large margin.
  • Efficiency: UGG runs significantly faster than PPCC, making it more practical for large-scale graph inference. For instance, it's noted to be five times faster on most protocols.
  • Effectiveness of adaptive gates: Ablation studies show that adaptively updating positive gates (aG) leads to the best performance, outperforming fixed gates (PG, PGcl).
  • Semi-supervised learning benefits: The paper demonstrates that UGG can effectively leverage unlabeled data in semi-supervised settings, leading to considerable improvements.

Limitations and Future Work

  • Limitations: The paper acknowledges that the EERR metric on IJB-S is lower than identification accuracy because it penalizes missed face detections, which is outside the scope of their work. The performance of negative gates is limited on the CSM dataset due to the lack of co-occurrence information.
  • Future Work: The authors suggest exploring the use of attribute information (e.g., gender) to construct more informative negative connections and adaptively update negative gates.

Summary updated

Abstract

Unconstrained video-based face recognition is a challenging problem due to significant within-video variations caused by pose, occlusion and blur. To tackle this problem, an effective idea is to propagate the identity from high-quality faces to low-quality ones through contextual connections, which are constructed based on context such as body appearance. However, previous methods have often propagated erroneous information due to lack of uncertainty modeling of the noisy contextual connections. In this paper, we propose the Uncertainty-Gated Graph (UGG), which conducts graph-based identity propagation between tracklets, which are represented by nodes in a graph. UGG explicitly models the uncertainty of the contextual connections by adaptively updating the weights of the edge gates according to the identity distributions of the nodes during inference. UGG is a generic graphical model that can be applied at only inference time or with end-to-end training. We demonstrate the effectiveness of UGG with state-of-the-art results in the recently released challenging Cast Search in Movies and IARPA Janus Surveillance Video Benchmark dataset.
BibTeX
@inproceedings{zheng2019uncertainty,
  author = {Jingxiao Zheng and Ruichi Yu and Jun-Cheng Chen and Boyu Lu and Carlos Domingo Castillo and Rama Chellappa},
  title = {{Uncertainty Modeling of Contextual-Connection between Tracklets for Unconstrained Video-based Face Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00079},
}
Figure