Relation-Aware Distribution Representation Network for Person Clustering with Multiple Modalities

Kaijian Liu, Shixiang Tang, Ziyue Li, Zhishuai Li, Lei Bai, Feng Zhu, Rui Zhao

Abstract

Person clustering with multi-modal clues, including faces, bodies, and voices, is critical for various tasks, such as movie parsing and identity-based movie editing. Related methods such as multi-view clustering mainly project multi-modal features into a joint feature space. However, multi-modal clue features are usually rather weakly correlated due to the semantic gap from the modality-specific uniqueness. As a result, these methods are not suitable for person clustering. In this paper, we propose a Relation-Aware Distribution representation Network (RAD-Net) to generate a distribution representation for multi-modal clues. The distribution representation of a clue is a vector consisting of the relation between this clue and all other clues from all modalities, thus being modality agnostic and good for person clustering. Accordingly, we introduce a graph-based method to construct distribution representation and employ a cyclic update policy to refine distribution representation progressively. Our method achieves substantial improvements of +6% and +8.2% in F-score on the Video Person-Clustering Dataset (VPCD) and VoxCeleb2 multi-view clustering dataset, respectively.

Introduction

Identifying and clustering characters in movies or TV shows is a complex task. A single person might appear as a face crop, a full-body shot, or just a voice clip. Traditionally, researchers try to project these different "modalities" into a single joint feature space. However, there is a massive semantic gap between these features. For example, body features are heavily influenced by clothing color, while face features are not; voice features are immune to lighting conditions, while visual features are not. Because these features are weakly correlated, forcing them into one space often leads to poor clustering performance. The authors argue that we shouldn't be looking at the features themselves, but at the relationships between them.

Methodology

The core innovation of the Relation-Aware Distribution representation Network (RAD-Net) is the Distribution Representation. Instead of a raw feature vector, each clue is represented by a vector $d(v_i)$ containing the probability that it shares an identity with every other clue in the graph. This makes the representation "modality-agnostic"—it doesn't matter if the input was a voice or a face; the output is just a set of identity relationships.

Overview of the RAD-Net framework showing how modality-specific clues (Face, Body, Voice) are transformed into modality-agnostic distribution representations.

To handle clues from different modalities, the network uses "bridge" nodes. If a face and a voice belong to the same video track, they are linked. The system can then infer the relationship between a face in Track A and a voice in Track B by "bridging" through other known associations. The model employs a Cyclic Update Policy to refine these distributions: it uses current features to estimate identity probabilities, then uses distribution similarity to enhance the features by aggregating information from likely neighbors, repeating the process to allow global identity information to refine local modality-specific features.

Experiments and Results

The model was evaluated on two major benchmarks: the Video Person-Clustering Dataset (VPCD), containing over 70,000 tracks from six movies/TV shows, and VoxCeleb2, a standard multi-view clustering dataset.

Quantitative comparison of RAD-Net against state-of-the-art unsupervised and supervised methods across multiple datasets.

RAD-Net significantly outperformed existing state-of-the-art methods. On the VPCD dataset, it achieved a +6% improvement in F-score over the previous best method (MuHPC). On the VoxCeleb2 dataset, it showed an +8.2% improvement in F-score on the 2048-identity test set. Qualitative results further demonstrate that RAD-Net can successfully retrieve body images when queried with a face image, a task where traditional feature-based methods often fail due to the modality gap.

Conclusion

RAD-Net demonstrates that focusing on relationships rather than raw feature fusion is a powerful strategy for multi-modal person clustering. By using distribution representations, the model effectively bridges the semantic gap between faces, bodies, and voices. Future work may address computational scaling for massive datasets and the integration of identity-based priors to further enhance clustering accuracy in varied video environments.

Summary updated

Abstract

Person clustering with multi-modal clues, including faces, bodies, and voices, is critical for various tasks, such as movie parsing and identity-based movie editing. Related methods such as multi-view clustering mainly project multi-modal features into a joint feature space. However, multi-modal clue features are usually rather weakly correlated due to the semantic gap from the modality-specific uniqueness. As a result, these methods are not suitable for person clustering. In this paper, we propose a Relation-Aware Distribution representation Network (RAD-Net) to generate a distribution representation for multi-modal clues. The distribution representation of a clue is a vector consisting of the relation between this clue and all other clues from all modalities, thus being modality agnostic and good for person clustering. Accordingly, we introduce a graph-based method to construct distribution representation and employ a cyclic update policy to refine distribution representation progressively. Our method achieves substantial improvements of +6% and +8.2% in F-score on the Video Person-Clustering Dataset (VPCD) and VoxCeleb2 multi-view clustering dataset, respectively. Codes will be released publicly upon acceptance.
BibTeX
@article{liu2024relation,
  author = {Kaijian Liu and Shixiang Tang and Ziyue Li and Zhishuai Li and Lei Bai and Feng Zhu and Rui Zhao},
  title = {{Relation-Aware Distribution Representation Network for Person Clustering with Multiple Modalities}},
  journal = {IEEE Trans. Multimedia},
  year = {2024},
  doi = {10.1109/tmm.2023.3304454},
}
Figure