Relation-Aware Distribution Representation Network for Person Clustering With Multiple Modalities

Kaijian Liu, Shixiang Tang, Ziyue Li, Zhishuai Li, Lei Bai, Feng Zhu, Rui Zhao

TL;DR

RAD-Net tackles the challenge of clustering people in videos by moving away from traditional feature-fusion methods. Instead, it represents each person-clue (face, body, or voice) as a "distribution representation"—a vector of its identity-relationships to every other clue in the dataset—effectively bypassing the "semantic gap" between different modalities.

What Problem Does This Paper Address?

Identifying and clustering characters in movies or TV shows is a complex task. A single person might appear as a face crop, a full-body shot, or just a voice clip. Traditionally, researchers try to project these different "modalities" into a single joint feature space.

However, there is a massive semantic gap between these features. For example, body features are heavily influenced by clothing color, while face features are not; voice features are immune to lighting conditions, while visual features are not. Because these features are weakly correlated, forcing them into one space often leads to poor clustering performance. The authors argue that we shouldn't be looking at the features themselves, but at the relationships between them.

Methods

The authors propose the Relation-Aware Distribution representation Network (RAD-Net). The core innovation is the Distribution Representation:

  1. Modality-Agnostic Vectors: Instead of a raw feature vector, each clue is represented by a vector $d(v_i)$ containing the probability that it shares an identity with every other clue in the graph. This makes the representation "modality-agnostic"—it doesn't matter if the input was a voice or a face; the output is just a set of identity relationships.
  2. Probabilistic Graph Inference: To handle clues from different modalities (e.g., a face vs. a voice), the network uses "bridge" nodes. If a face and a voice belong to the same video track, they are linked. The system can then infer the relationship between a face in Track A and a voice in Track B by "bridging" through other known associations.
  3. Cyclic Update Policy: The model doesn't just calculate these distributions once. It uses a cyclic process:
  4. Step 1: Use current features to estimate the distribution (identity probabilities).
  5. Step 2: Use the distribution similarity to "enhance" the features by aggregating information from likely neighbors.
  6. Step 3: Repeat. This allows global identity information to refine local modality-specific features.
  7. Density-Aware Sampling: To keep the graph computation manageable, the authors use a sampling strategy that selects the most representative and "hard" examples from person tracks.

Datasets

The model was evaluated on two major benchmarks:

  • Video Person-Clustering Dataset (VPCD): A large-scale dataset containing 32,999 face tracks, 36,724 body tracks, and 9,863 voice tracks from six movies/TV shows (e.g., Sherlock, Friends, The Big Bang Theory).
  • VoxCeleb2: A standard multi-view clustering dataset used here to test the model's ability to handle face and audio features specifically.

Key Insights and Ideas

  • Relationships > Features: The "aha moment" is realizing that while a face and a voice look/sound nothing alike, their pattern of relationships to the rest of the world is identical if they belong to the same person.
  • The Power of the Bridge: By using the "track" (a clip where a face and body are physically linked) as a bridge, the model can propagate identity information across the semantic gap without needing a complex joint-embedding space.
  • Robustness to Noise: The distribution representation is surprisingly resilient. Even when the authors intentionally introduced "noisy associations" (e.g., linking a face to the wrong body), the model's performance only dropped slightly because it relies on global relations rather than single-point matches.

Key Results

RAD-Net significantly outperformed existing unsupervised and supervised state-of-the-art methods:

  • VPCD Dataset: Achieved a +6% improvement in F-score over the previous best method (MuHPC).
  • VoxCeleb2 Dataset: Achieved an +8.2% improvement in F-score on the 2048-identity test set.
  • Qualitative Success: Visualization shows that RAD-Net can successfully retrieve body images when queried with a face image, a task where traditional feature-based methods often fail due to the modality gap.

Limitations and Future Work

  • Computational Scaling: While the authors claim the complexity is "linearithmic" due to K-Nearest Neighbor searches, building and updating a global relationship graph for millions of tracks remains a significant engineering challenge.
  • Fixed Sampling: The current method relies on a fixed number of sampled features per track. In very long videos with highly varied appearances, this might miss critical "hard" examples.
  • Future Directions: The authors suggest incorporating user preferences or identity-based priors into the graph to allow for interactive or few-shot person clustering.

Summary updated

Abstract

Person clustering with multi-modal clues, including faces, bodies, and voices, is critical for various tasks, such as movie parsing and identity-based movie editing. Related methods such as multi-view clustering mainly project multi-modal features into a joint feature space. However, multi-modal clue features are usually rather weakly correlated due to the semantic gap from the modality-specific uniqueness. As a result, these methods are not suitable for person clustering. In this article, we propose a R elation- A ware D istribution representation Network (RAD-Net) to generate a distribution representation for multi-modal clues. The distribution representation of a clue is a vector consisting of the relation between this clue and all other clues from all modalities, thus being modality agnostic and good for person clustering. Accordingly, we introduce a graph-based method to construct distribution representation and employ a cyclic update policy to refine distribution representation progressively. Our method achieves substantial improvements of +6% and +8.2% in F-score on the Video Person-Clustering Dataset (VPCD) and VoxCeleb2 multi-view clustering dataset, respectively.
BibTeX
@article{liu2024relation,
  author = {Kaijian Liu and Shixiang Tang and Ziyue Li and Zhishuai Li and Lei Bai and Feng Zhu and Rui Zhao},
  title = {{Relation-Aware Distribution Representation Network for Person Clustering With Multiple Modalities}},
  journal = {IEEE Trans. Multimedia},
  year = {2024},
  doi = {10.1109/tmm.2023.3304454},
}

Loading related content...

Figure