Deep Mean Maps

Junier B. Oliva, Danica J. Sutherland, Barnabás Póczos, Jeff G. SchneiderarXiv 2015 · OpenReview

TL;DR

This paper introduces Deep Mean Maps (DMMs), a novel framework that jointly learns high-level features in deep neural networks and represents their distributions non-parametrically. By integrating a "mean map layer" into existing CNN architectures, DMMs improve image classification performance, especially in sample-efficient scenarios.

What Problem Does This Paper Address?

Modern computer vision heavily relies on two powerful techniques: deep learning architectures (like Convolutional Neural Networks or CNNs) for automatically learning features, and distribution-based methods for representing data. While both have seen great success individually, their joint application has been limited. Deep architectures typically output a fixed-size vector of high-level features, which are then fed into a classifier. This approach often treats these features as independent, ignoring their underlying distributional properties within an image. This paper aims to bridge this gap by developing a method to leverage the distributions of these learned high-level features for improved classification.

Methods

The core of the DMM framework is the mean map layer. This layer is designed to be seamlessly integrated into existing CNN architectures. It takes the top-level convolutional features of an image and computes a mean map embedding. This embedding is a non-parametric representation of the distribution of these features.

Here's a breakdown of the key components:

  • Mean Map Embeddings (MMEs): MMEs are a technique to represent probability distributions as vectors in a Reproducing Kernel Hilbert Space (RKHS). This allows familiar kernel methods (like Support Vector Machines) to be applied to distributions. The paper uses a specific formulation of MMEs.
  • Random Feature Embeddings: To make MMEs scalable for large datasets and high-dimensional features, the authors employ random feature embeddings. This technique approximates kernel evaluations using finite-dimensional dot products, avoiding the computationally expensive N x N Gram matrix.
  • Mean Map Layer Implementation: The mean map layer is implemented using standard CNN operations:
  • Frequency Convolution: A set of 1x1 convolutions are applied, each associated with a learned frequency.
  • Cosine Layer: An element-wise cosine non-linearity is applied to the results of the frequency convolutions.
  • Global Average Pooling: The output is then pooled globally across spatial dimensions to produce the final mean map embedding.
  • Joint Learning: Crucially, this mean map layer can be trained end-to-end with the rest of the CNN. This means that not only are the distributions of features used for classification, but the features themselves are also tuned to be more discriminative based on their distributional properties.

The paper also explores different ways to integrate this mean map layer into existing CNNs: "Replacing" (replacing fully connected layers with the mean map layer), "Replicating" (feeding features to both the mean map layer and other layers), and "Forking" (creating distinct feature paths for the mean map layer and other layers).

Datasets

The paper evaluates DMMs on the following datasets:

  • Synthetic Data: A custom dataset generated from the Colored Brodatz Texture (CBT) database to specifically test the ability of DMMs to uncover visual distributional patterns.
  • Flickr Style: A dataset of 80,000 images across 20 style categories.
  • Wikipaintings: A dataset of approximately 83,000 images across 25 art genres.
  • Places-205: A large-scale dataset with 2.4 million training images across 205 scene categories.

The authors fine-tune pre-trained networks (CaffeNet and GoogleNet) on these datasets.

Key Insights and Ideas

  • Distributions Matter: The paper strongly suggests that the distributional characteristics of high-level features, not just their individual values, are crucial for robust image classification.
  • Sample Efficiency: DMMs demonstrate remarkable performance even with very limited training data (e.g., 10 instances per class), outperforming standard CNNs. This highlights the power of learning from distributions when data is scarce.
  • Seamless Integration: The mean map layer is designed to be a plug-and-play component for existing CNNs, requiring only standard operations and allowing for end-to-end training.
  • Theoretical Foundation: The use of MMEs provides a theoretical grounding, allowing the interpretation of operations within the DMM framework as RKHS inner products on distributional embeddings.

Key Results

  • On the synthetic dataset, DMMs were able to almost perfectly distinguish classes with only 10 training instances per class, significantly outperforming baseline CNNs.
  • When adapting pre-trained CaffeNet and GoogleNet models to the Flickr Style and Wikipaintings datasets, DMM extensions (particularly the "Replacing" variant) consistently outperformed the "base" models on both validation and test sets.
  • On the large Places-205 dataset, DMM extensions also showed improvements over the base models, demonstrating scalability and effectiveness on a challenging real-world task. For example, one DMM variant achieved 46.62% Top-1 accuracy on Places-205 with CaffeNet, compared to 45.69% for the base model.

Limitations and Future Work

  • Frequency Learning: While the paper mentions the possibility of learning the frequencies (ωd) in the mean map layer, the primary experiments use fixed, randomly drawn frequencies. Learning these frequencies could potentially lead to further improvements.
  • Kernel Choice: The paper focuses on a specific formulation of MMEs. Exploring other kernel choices or MME formulations might yield different results.
  • Computational Cost: While random features make the approach scalable, the mean map layer does add some computational overhead compared to a standard CNN without it. The paper states that training time is comparable, but further analysis on inference speed might be beneficial.
  • Future Work: The authors suggest exploring the learned frequencies and potentially applying DMMs to other vision tasks beyond classification. The flexibility of the framework also opens doors for various architectural extensions.

Summary updated

Abstract

The use of distributions and high-level features from deep architecture has become commonplace in modern computer vision. Both of these methodologies have separately achieved a great deal of success in many computer vision tasks. However, there has been little work attempting to leverage the power of these to methodologies jointly. To this end, this paper presents the Deep Mean Maps (DMMs) framework, a novel family of methods to non-parametrically represent distributions of features in convolutional neural network models. DMMs are able to both classify images using the distribution of top-level features, and to tune the top-level features for performing this task. We show how to implement DMMs using a special mean map layer composed of typical CNN operations, making both forward and backward propagation simple. We illustrate the efficacy of DMMs at analyzing distributional patterns in image data in a synthetic data experiment. We also show that we extending existing deep architectures with DMMs improves the performance of existing CNNs on several challenging real-world datasets.
BibTeX
@misc{oliva2015deep,
  author = {Junier B. Oliva and Danica J. Sutherland and Barnabás Póczos and Jeff G. Schneider},
  title = {{Deep Mean Maps}},
  year = {2015},
  eprint = {1511.04150},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.04150},
}
Figure