Graphical Model-Based Learning in High Dimensional Feature Spaces

Zhao Song, Yuke ZhuAAAI 2013 · OpenReview

Abstract

Digital media tend to combine text and images to express richer information, especially on image hosting and online shopping websites. This trend presents a challenge in understanding the contents from different forms of information. Features representing visual information are usually sparse in high dimensional space, which makes the learning process intractable. In order to understand text and its related visual information, we present a new graphical model-based approach to discover more meaningful information in rich media. We extend the standard Latent Dirichlet Allocation (LDA) framework to learn in high dimensional feature spaces.

Introduction

In modern digital media environments, such as Flickr or e-commerce platforms like Amazon, information is inherently multimodal, combining textual descriptions with visual content. While Latent Dirichlet Allocation (LDA) has become a standard for topic modeling in text, its application to visual data is non-trivial. Visual features are typically continuous, sparse, and reside in high-dimensional spaces. Standard LDA, designed for discrete word tokens, often fails to capture the correlations between modalities when applied to raw visual feature vectors due to the "curse of dimensionality." This paper addresses these challenges by proposing a hierarchical graphical model that bridges the gap between text and high-dimensional visual features.

Methods

The proposed approach extends the LDA framework by introducing a hierarchical cluster layer that generates both Gaussian distributions for images and topics for text. This structure treats a document as a combination of a cluster, a visual Gaussian distribution, and a set of textual topics.

The Generative Process

The model follows a structured generative process to produce multimodal documents:

  1. Cluster Selection: A latent cluster $c$ is drawn from a distribution of cluster importance, parameterized by $\alpha$.
  2. Visual Path: A Gaussian distribution $g$ is drawn from the cluster (influenced by $\beta$). A visual feature vector $l$ is then sampled from this Gaussian, defined by mean $\mu$ and covariance $\Sigma$.
  3. Textual Path: A topic $z$ is drawn based on the cluster-specific topic distribution $\phi_c$. Finally, words $w$ are sampled from the topic-word distribution $\theta_z$.
The proposed graphical model illustrating the hierarchical relationship between clusters, topics, and visual features.

Key Innovations

  • Shared Latent Clusters: By conditioning both visual Gaussians and textual topics on a shared "cluster" variable, the model explicitly discovers semantic correlations between the two modalities.
  • Optimized Dimension Reduction: To manage high-dimensional visual noise, the authors integrate Principal Component Analysis (PCA) directly into the learning loop. The truncation point ($k$) is optimized using perplexity-based gradient descent, ensuring the model retains only the most informative visual components.
  • Image Representation: The system employs a "Bag-of-Visual-Words" approach, using SIFT descriptors and k-means clustering to represent images as histograms suitable for the graphical model.

Evaluation

The model was evaluated on two real-world datasets: Social20 (20 visual concepts from Flickr) and the Attribute Discovery Dataset (product images and descriptions from like.com). The primary evaluation metric was perplexity, which measures the model's predictive power on unseen samples.

The proposed model was compared against standard text-based LDA and a regularized LDA variant that incorporates image vectors. Results across textual, visual, and combined perplexity dimensions demonstrated that the hierarchical approach effectively learns in high-dimensional spaces where traditional frameworks struggle.

Discussion and Conclusion

The "Cluster" node serves as a critical semantic bridge. For instance, a cluster representing "Sunsets" can simultaneously guide the model toward warm-colored Gaussian distributions in the visual space and relevant terms like "orange" or "evening" in the textual space. This formalization enables robust cross-modal information retrieval.

While the model shows promise, it faces limitations regarding computational overhead during Expectation-Maximization (EM) steps and its reliance on traditional features like SIFT. Future work could involve adapting the framework for modern deep learning embeddings (e.g., from CNNs or Transformers) and extending the model to other media types such as audio or video.

Summary updated

Abstract

Digital media tend to combine text and images to express richer information, especially on image hosting and online shopping websites. This trend presents a challenge in understanding the contents from different forms of information. Features representing visual information are usually sparse in high dimensional space, which makes the learning process intractable. In order to understand text and its related visual information, we present a new graphical model-based approach to discover more meaningful information in rich media. We extend the standard Latent Dirichlet Allocation (LDA) framework to learn in high dimensional feature spaces.
BibTeX
@inproceedings{song2013graphical,
  author = {Zhao Song and Yuke Zhu},
  title = {{Graphical Model-Based Learning in High Dimensional Feature Spaces}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2013},
  url = {https://openreview.net/forum?id=BJNXOCl_Wr},
}
Figure