Compositional Mixture Representations for Vision and Text

Stephan Alaniz, Marco Federici, Zeynep Akata

Abstract

Learning meaningful visual representations often requires extensive manual labeling. We present CoMix, a framework that leverages the compositional structure of language to guide the discovery of objects in images. By modeling both modalities as Gaussian Mixture Models (GMMs), CoMix achieves weakly supervised object detection and robust image retrieval without spatial supervision.

Introduction

In computer vision, the gold standard for representation learning typically involves massive datasets annotated with bounding boxes. While unsupervised methods exist, they frequently fail to distinguish salient objects from background noise. Humans, by contrast, utilize language as a structural prior; hearing "a cat and a frog" directs our attention to two distinct entities. CoMix addresses this by enforcing compositional representation learning, decomposing complex scenes into constituent parts guided solely by text captions.

The CoMix Framework

The core innovation of CoMix is treating the latent space as a Gaussian Mixture Model (GMM), facilitating alignment between discrete linguistic concepts and image patches.

The CoMix architecture: (a) Image and text encoders map inputs to GMMs in a shared latent space, aligned via KL-divergence. (b) The corresponding graphical model.
  1. Text Encoder: Words in a caption are mapped to individual Gaussian distributions. A sentence is represented as a mixture of these Gaussians, where each component corresponds to a specific concept.
  2. Image Encoder: Utilizing a Spatial Transformer Network (STN), the model predicts $k$ sets of transformation parameters to "crop" patches from the image. Each patch is encoded into its own Gaussian distribution.
  3. Alignment via KL-Divergence: Training minimizes the KL-divergence between the image-based and text-based GMMs. This forces the network to find image patches that correspond to the words present in the text.
  4. Image Area Loss: A penalty term encourages predicted bounding boxes to be as small as possible, preventing the model from defaulting to global image views.
  5. Text Decoder: A decoder reconstructs text labels from the latent GMM to ensure the representation captures semantic information.

Experimental Setup

Evaluation was conducted on two synthetic multi-object datasets:

  • MultiMNIST: $56 \times 56$ canvases with 1-4 handwritten digits.
  • MultiCIFAR10: $80 \times 80$ canvases with 1-4 objects from CIFAR10 classes. These datasets provide ground-truth locations for evaluation, though these are withheld during training.

Key Results and Insights

CoMix demonstrates significant improvements in weakly supervised object detection (WSOD) and interpretability.

Qualitative results on MultiMNIST and MultiCIFAR10. The model successfully identifies and localizes multiple objects (left), with the middle and right columns showing the attention and classification probabilities for each component.
  • Weakly Supervised Object Detection: On MultiCIFAR10, CoMix achieved 75.83% mAP, vastly outperforming the WSDDN baseline (49.47%).
  • Hard Attention for Interpretability: The use of an STN makes the model's internal logic transparent; the extracted patches explicitly show which regions the model associates with specific labels.
  • Combinatorial Extrapolation: By learning "parts" rather than "wholes," CoMix generalizes to novel object combinations not seen during training.
  • Emergent Counting: The model naturally learns to represent multiple instances of the same class, reflecting the quantity in the joint classification probability.

Conclusion and Future Directions

CoMix proves that linguistic structure can serve as a powerful prior for visual disentanglement. Future work will focus on scaling this approach to natural language captions using large-scale language models and adapting the architecture to handle the complexities of real-world, cluttered environments.

Summary updated

Abstract

Learning a common representation space between vision and language allows deep networks to relate objects in the image to the corresponding semantic meaning. We present a model that learns a shared Gaussian mixture representation imposing the compositionality of the text onto the visual domain without having explicit location supervision. By combining the spatial transformer with a representation learning approach we learn to split images into separately encoded patches to associate visual and textual representations in an interpretable manner. On variations of MNIST and CIFAR10, our model is able to perform weakly supervised object detection and demonstrates its ability to extrapolate to unseen combination of objects.
BibTeX
@inproceedings{alaniz2022compositional,
  author = {Stephan Alaniz and Marco Federici and Zeynep Akata},
  title = {{Compositional Mixture Representations for Vision and Text}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
}
Figure