Temporally Coherent Chinese Restaurant Process for Discovery of Persons and Corresponding Tracklets from User-generated Videos

Adway Mitra, Soma Biswas, Chiranjib BhattacharyyaarXiv 2014 · OpenReview

Abstract

A video can be represented as a sequence of tracklets, each spanning 10-20 frames, and associated with one entity (e.g., a person). The task of Entity Discovery in videos can be naturally posed as tracklet clustering. We approach this task by leveraging Temporal Coherence (TC): the fundamental property of videos that each tracklet is likely to be associated with the same entity as its temporal neighbors. Our major contributions are the first Bayesian nonparametric models for TC at the tracklet-level. We extend the Chinese Restaurant Process (CRP) to propose TC-CRP, and further to the Temporally Coherent Chinese Restaurant Franchise (TC-CRF) to jointly model short temporal segments. On the task of discovering persons in TV serial videos without metadata like scripts, these methods show considerable improvement in cluster purity and person coverage compared to state-of-the-art approaches to tracklet clustering. The proposed methods can perform online tracklet clustering on streaming videos with little performance deterioration and can automatically reject tracklets resulting from false detections.

Introduction

Identifying "who is who" in a video is a trivial task for humans but remains a significant challenge for machine learning, particularly in unscripted "videos in the wild" (like YouTube uploads) that lack metadata. The standard approach involves tracklet clustering—linking short, reliable sequences of detections (tracklets) across an entire video. Traditional clustering algorithms often struggle because they require a predefined number of entities, fail to handle "junk" detections (false positives from face detectors), and ignore the inherent temporal logic of video: an entity cannot be in two places at once, yet an entity appearing now is highly likely to be the same one seen moments ago.

Methodology

This research introduces two novel Bayesian nonparametric models designed to exploit temporal coherence as a primary cue for clustering.

TC-CRP: Temporally Coherent Chinese Restaurant Process

TC-CRP builds upon the Chinese Restaurant Process, which allows the number of clusters (entities) to grow dynamically as more data is processed. It incorporates spatio-temporal constraints: a "must-link" logic for temporally proximal tracklets and a "don't-link" logic for overlapping tracklets (since one entity cannot have two tracklets simultaneously). Additionally, it includes a specialized "Component 0"—a high-variance cluster designed to capture and isolate false detections, preventing them from polluting the entity clusters.

TC-CRF: Temporally Coherent Chinese Restaurant Franchise

TC-CRF extends the model to handle video segments (such as scenes or shots). By utilizing an Indian Buffet Process (IBP) prior, it models the sparse appearance of entities, reflecting the reality that only a small subset of the total cast appears in any given scene. This sparsity improves accuracy in long-form content where characters frequently enter and exit the narrative.

Figure 2: Visualization of tracklet clustering across video frames. The colored boxes represent discovered entities tracked over time, illustrating how the model maintains identity across a sequence.

Both models support online inference, enabling real-time tracklet clustering as video streams, which is a significant advantage over existing approaches that require the entire video file upfront.

Experimental Evaluation

The models were evaluated on diverse real-world datasets:

  • TV Series: Three episodes of The Big Bang Theory.
  • Long-form Drama: Six episodes of the Indian series The Mahabharata, characterized by low video quality, motion blur, and complex character interactions.
  • Non-Person Entities: Videos of cars and airplanes from the YouTube Objects dataset to demonstrate generalizability beyond human faces.

Results and Insights

The TC-CRF model consistently outperformed state-of-the-art baselines (such as WBSLRR and sHDP-HMM) across several metrics:

  • Cluster Purity: Achieved scores as high as 0.97, indicating highly accurate entity grouping.
  • Entity Coverage: Successfully identified the vast majority of prominent characters (e.g., 21 out of 24 in a complex Mahabharata episode).
  • Online Performance: Demonstrated minimal performance degradation during streaming tests compared to offline processing, making the system viable for real-time applications.

Key insights include the utility of temporal coherence as a "free" label and the effectiveness of treating noise as a distinct, high-variance cluster. Furthermore, the discovery of entities enables entity-driven video summarization, creating concise summaries based on the presence of specific characters rather than just visual importance.

Conclusion and Future Work

While the proposed models significantly advance entity discovery, challenges remain. Significant pose variations (e.g., a character turning from profile to front-facing) can still lead to over-clustering. Future work will explore integrating more robust feature representations, such as deep-learning-based face embeddings, and improving the resilience of the initial tracklet generation process.

Summary updated

Abstract

A video can be represented as a sequence of tracklets, each spanning 10-20 frames, and associated with one entity (eg. a person). The task of \emph{Entity Discovery} in videos can be naturally posed as tracklet clustering. We approach this task by leveraging \emph{Temporal Coherence}(TC): the fundamental property of videos that each tracklet is likely to be associated with the same entity as its temporal neighbors. Our major contributions are the first Bayesian nonparametric models for TC at tracklet-level. We extend Chinese Restaurant Process (CRP) to propose TC-CRP, and further to Temporally Coherent Chinese Restaurant Franchise (TC-CRF) to jointly model short temporal segments. On the task of discovering persons in TV serial videos without meta-data like scripts, these methods show considerable improvement in cluster purity and person coverage compared to state-of-the-art approaches to tracklet clustering. We represent entities with mixture components, and tracklets with vectors of very generic features, which can work for any type of entity (not necessarily person). The proposed methods can perform online tracklet clustering on streaming videos with little performance deterioration unlike existing approaches, and can automatically reject tracklets resulting from false detections. Finally we discuss entity-driven video summarization- where some temporal segments of the video are selected automatically based on the discovered entities.
BibTeX
@misc{mitra2014temporally,
  author = {Adway Mitra and Soma Biswas and Chiranjib Bhattacharyya},
  title = {{Temporally Coherent Chinese Restaurant Process for Discovery of Persons and Corresponding Tracklets from User-generated Videos}},
  year = {2014},
  eprint = {1409.6080},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1409.6080},
}
Figure