A Topic Model Approach to Multi-Modal Similarity

Rasmus Troelsgård, Bjørn Sand Jensen, Lars Kai HansenarXiv 2014 · arXiv

Introduction

In modern multimedia applications, objects like songs are defined by diverse data types, including audio, lyrics, and social tags. The challenge of "multi-modal fusion" involves combining these disparate sources into a single similarity metric without one modality—such as high-dimensional audio—overwhelming others. This paper proposes a multi-modal Latent Dirichlet Allocation (LDA) framework to address this, ensuring a consistent "geometric" representation of similarity across heterogeneous data.

Methodology: Multi-Modal LDA

The core of the proposed approach is a variant of Multi-modal LDA. In this model, each document (e.g., a song) is represented by a single, shared distribution over topics ($\theta$). Crucially, each topic is associated with multiple multinomial distributions, one for each modality. This structure forces the model to find a symmetric latent space that simultaneously explains all data types.

Plate notation for the multi-modal LDA model, illustrating the shared topic distribution $\theta$ across modalities.

To measure similarity between objects, the authors utilize Predictive Likelihood rather than standard topic distribution comparisons. Similarity is defined as the probability of observing the features of one song given the topic profile of another. This method is more robust to stochastic variations in topic assignments, as it focuses on the underlying vocabulary generated by the topics.

Evaluation and Stability

A significant concern with topic models is their stochastic nature; different initializations can lead to different latent topics. To validate model consistency, the authors employ the Mantel Test, a non-parametric permutation test, to compare similarity matrices generated from different runs or data sources.

The model was evaluated using a subset of the Million Song Dataset (MSD), comprising 30,000 tracks. Modalities included user tags, lyrics, artist tags, listening history, and vector-quantized audio features.

Key Results

The results demonstrate high model stability. The Spearman correlation between similarity matrices from different random initializations was consistently high, often exceeding 0.9 for the audio modality.

Spearman's correlation coefficients across different numbers of topics, demonstrating high stability in similarity structures.

Furthermore, the model showed significant cross-modality consistency. Even when comparing models trained on audio versus metadata, the Mantel test rejected the null hypothesis of no correlation (p < 0.01), suggesting that the model captures a fundamental underlying similarity. Interestingly, as the number of topics ($T$) increases, the correlation between different modality groups tends to decrease, indicating that higher topic counts may lead to over-specialization to specific data types.

Conclusion and Future Work

While the predictive likelihood measure provides robustness, it is computationally expensive, potentially limiting real-time application. Future work could explore more explicit relational modeling between modalities and investigate how different data types contribute to human perceptions of music similarity. This framework offers a statistically grounded method for fusing multi-modal data into a stable, unified similarity space.

Summary updated

BibTeX
@misc{troelsgard2014topic,
  author = {Rasmus Troelsgård and Bjørn Sand Jensen and Lars Kai Hansen},
  title = {{A Topic Model Approach to Multi-Modal Similarity}},
  year = {2014},
  eprint = {1405.6886},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1405.6886},
}
Figure