VISTA: Vision-Language Integrated Semantic Triage Architecture for Online Face Clustering in Long Videos

Jiashu Liao, Joanne Garde-Hansen, Sanjay Sharma, Tanaya GuhaarXiv 2026 · arXiv

TL;DR

VISTA is the first vision-language framework designed for online face clustering in long, unconstrained videos. By integrating semantic attributes (like age, gender, and clothing) from Vision-Language Models (VLMs) with traditional geometric embeddings, VISTA significantly improves identity tracking through a novel "semantic triage" system that resolves visual ambiguities without sacrificing real-time efficiency.

What Problem Does This Paper Address?

Identifying and grouping faces by identity across hours of video is a foundational task for automated narrative analysis and video browsing. Most current state-of-the-art methods are "offline," meaning they require the entire video to be available upfront for global optimization. However, real-world streaming and large-scale processing require "online" algorithms that process video in a single pass.

Existing online methods, such as Constrained Online Face Clustering (COFC), rely almost exclusively on visual geometric features (e.g., FaceNet embeddings). These fail spectacularly when characters encounter extreme lighting changes, dramatic pose variations, or costume shifts—common occurrences in long-form media like TV series. While VLMs like CLIP can provide helpful semantic cues, they are prone to "hallucinations" in poor lighting and can be misled by costume changes, often leading to "over-segmentation," where a single person is split into dozens of tiny, fragmented clusters.

Methods

VISTA (Vision-Language Integrated Semantic Triage Architecture) introduces a three-stage pipeline to bridge the gap between low-level visual features and high-level semantic reasoning.

Figure 1: The VISTA pipeline. Stage 1 extracts geometric and VLM features. Stage 2 forms intra shot tracks. Stage 3 performs inter shot association via SSR mapping and dynamic three zone semantic triage.

The core innovation lies in Stage 3: Inter-Shot Semantic Association, which uses three key modules:

  1. Symbolic Semantic Representation (SSR): Raw VLM predictions are often jittery (e.g., oscillating between "Age: 20-29" and "Age: 30-39"). SSR maps these noisy labels to stable, discrete tokens (e.g., <age_adult>) via a deterministic registry, acting as a temporal denoising layer.
  2. Three-Zone Semantic Triage: Rather than applying semantic rules to every face, VISTA uses a Gaussian Mixture Model (GMM) to dynamically partition face-track pairs into three zones based on visual similarity:
  3. High-confidence Zone: Direct merge based on visual similarity.
  4. Ambiguity Zone: Invokes full semantic arbitration to resolve uncertainty.
  5. Reject Zone: Direct rejection, unless persistent semantic cues (gender/age) strongly suggest a "rescue" merge.
  6. Spatio-Temporal Decoupled Arbitration: VISTA treats attributes differently based on their stability. Persistent traits (gender, age) are weighted heavily for long-range association, while transient traits (clothing, hair) are only used for short temporal gaps to avoid errors caused by costume changes. Additionally, a "low-confidence masking" mechanism attenuates semantic penalties during dark or blurry scenes to prevent VLM hallucinations from corrupting the clusters.
Figure 2: Detailed mechanisms of VISTA. (a) Three Zone Semantic Triage with GMM dynamic thresholding. (b) SSR maps raw predictions to stable tokens. (c) Semantic Arbitration via temporal voting compares symbolic profiles.

Datasets

The authors evaluated VISTA on the VPCD benchmark dataset, which includes 15 full episodes from three distinct TV series:

  • The Big Bang Theory (TBBT): Well-lit sitcom, relatively stable environments.
  • Buffy the Vampire Slayer: Dark action scenes with significant illumination challenges.
  • Sherlock: Complex thriller featuring rich narratives, frequent costume changes, and disguises.

VISTA was compared primarily against COFC, the current state-of-the-art for online, unsupervised face clustering.

Key Insights and Ideas

  • Selective Arbitration is Key: You don't need a VLM for every frame. By only invoking semantic reasoning for "ambiguous" cases (about 20% of tracklets), VISTA maintains a massive throughput of over 1800 FPS for its clustering module.
  • Denoising via Abstraction: Mapping continuous or fine-grained VLM outputs to coarse "symbols" (SSR) makes the system significantly more robust to the frame-level noise inherent in zero-shot models.
  • Decoupling Space and Time: Recognizing that clothing changes over time but gender does not allows the model to maintain identity continuity across long temporal gaps where visual similarity might otherwise fail.
  • Demographic Awareness: The authors noted that VLM intervention rates vary by demographic (e.g., children trigger more interventions), suggesting that semantic triage can help highlight and potentially mitigate biases in underlying vision models.
Figure 4: Distribution of VLM intervention and rejection rates across different demographic groups.

Key Results

VISTA consistently outperformed the COFC baseline across all metrics:

  • Buffy: Achieved a massive +18.51 percentage point improvement in Identification Accuracy (ID_Acc) over COFC, proving its robustness in dark, challenging environments.
  • TBBT: Improved Weighted Cluster Purity (WCP) from 92.13% to 96.97%.
  • Efficiency: While the end-to-end pipeline (including detection and feature extraction) runs at 7–8 FPS, the core VISTA clustering logic is extremely lightweight, operating at 1800–2300 FPS.

Qualitatively, VISTA produces much cleaner timelines. In Sherlock, where characters like Lestrade and Mycroft look visually similar, COFC often confuses their identities. VISTA uses semantic cues to keep these clusters distinct.

Figure 3: Timeline level qualitative comparison on Sherlock. Top: ground truth; Middle: COFC; Bottom: VISTA. Each row represents a character, with colors indicating predicted clusters. Compared to COFC’s cross identity confusion, VISTA maintains cleaner global identity organization and better separates visually similar characters (e.g., Lestrade and Mycroft).

Limitations and Future Work

While VISTA excels at maintaining purity, it faces a "purity-continuity trade-off" in extremely volatile datasets like Sherlock. In some cases, VISTA's conservative semantic triage leads to over-segmentation (fragmenting one identity into multiple clusters) rather than mixing identities.

Future work aims to address this through:

  • Adaptive Smoothing: Developing decay rates that adapt to the specific volatility of a scene.
  • Calibrated Uncertainty: Better quantifying when a VLM is likely to be "hallucinating."
  • Expanded Vocabularies: Moving beyond fixed symbolic registries to more flexible semantic representations.

Summary updated

Abstract

Clustering faces by identity in long, continuous video streams is a challenging task that serves as a building block for human-centric video analysis. The majority of existing face clustering models are offline, i.e., they assume the entire data to be available altogether. In many practical scenarios, however, the entire data is either unavailable or poses computational challenges to be processed in its entirety. A handful of online methods that exist rely on geometric visual features and simple thresholding techniques, which fail when the faces exhibit significant appearance variations in long, unconstrained videos. To this end, we present VISTA, the first vision language framework for online face clustering in long videos. VISTA leverages semantic reasoning in a controlled, uncertainty aware fashion to maintain both high cluster purity and long range identity continuity. This makes it robust to the variations in illumination, pose, and costume changes that the faces and characters exhibit in long real world videos. Extensive experiments on the VPCD benchmark dataset (with 15 full episodes from The Big Bang Theory [2007-2019], Buffy the Vampire Slayer [1997-2003], and Sherlock [2010-2017]) demonstrate that VISTA achieves a robust precision and recall balance, consistently improving over the strongest matched online unsupervised baseline (COFC) under identical detector, embeddings, and evaluation protocol. Code and supplementary material: https://github.com/SkyJiashu/VISTA.
BibTeX
@misc{liao2026vista,
  author = {Jiashu Liao and Joanne Garde-Hansen and Sanjay Sharma and Tanaya Guha},
  title = {{VISTA: Vision-Language Integrated Semantic Triage Architecture for Online Face Clustering in Long Videos}},
  year = {2026},
  eprint = {2611.04932},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04932},
}
Figure