E$^2$BoWs: An End-to-End Bag-of-Words Model via Deep Convolutional Neural Network

Xiaobin Liu, Shiliang Zhang, Tiejun Huang, Qi TianarXiv 2017 · arXiv

Abstract

Large-scale image retrieval requires both high semantic accuracy and computational efficiency. Traditional Bag-of-Words (BoW) models offer efficiency through inverted indexing but suffer from the "semantic gap" due to hand-crafted features. Conversely, Deep Convolutional Neural Networks (DCNNs) provide rich semantic representations but typically produce dense vectors that are expensive to search. This paper introduces E$^2$BoWs, an end-to-end trainable framework that integrates BoW logic into a DCNN. By generating sparse "visual words" from semantic feature maps, E$^2$BoWs achieves state-of-the-art retrieval accuracy while maintaining a low memory footprint and high search speed.

Introduction

In the domain of large-scale image search, a persistent trade-off exists between the deep semantic understanding of modern neural networks and the lightning-fast efficiency of classical Bag-of-Words (BoW) models. Classical BoW models rely on hand-crafted features like SIFT, which struggle to capture high-level concepts—a phenomenon known as the "semantic gap." While deep learning features have largely bridged this gap, they are typically dense and computationally expensive to query at scale.

Researchers from Peking University and UT San Antonio have proposed E$^2$BoWs (End-to-End Bag-of-Words) to resolve this tension. This model integrates BoW logic directly into a Deep Convolutional Neural Network (DCNN), creating a system that is semantically robust, sparse, and optimized for massive retrieval tasks.

The E$^2$BoWs Framework

The E$^2$BoWs architecture, based on a modified GoogLeNet, transforms deep convolutional features into sparse visual words through a unified pipeline.

The E$^2$BoWs end-to-end architecture, illustrating the generation of semantic feature maps and the Bag-of-Words layer.

Semantic Feature Map (SFM) Generation

Instead of utilizing a standard fully connected (FC) layer for classification, E$^2$BoWs transforms the final FC layer into a convolutional layer. This produces $n$ feature maps (corresponding to $n$ categories) that preserve spatial information. Unlike global vectors, these maps retain data regarding the location and scale of objects within the image.

Bag-of-Words Layer (BoWL)

The SFMs are passed into a specialized Bag-of-Words Layer. This layer generates $m$ visual words for each map, resulting in a total potential vocabulary of $m \times n$ words. To ensure efficiency, the model employs Dynamic Thresholding, learning a parameter ($\beta$) to prune noisy visual words with low response values.

Training Objectives

The model is optimized using a three-component loss function:

  1. Classification Loss: Drives the model to learn discriminative features for object identification.
  2. Triplet Similarity Loss: Employs Cosine distance to ensure similar images share visual words while dissimilar ones do not.
  3. Sparsity Loss: Uses KL-divergence to force the majority of visual words to zero, enabling the use of efficient inverted indices.

Experimental Results

The researchers evaluated E$^2$BoWs across several benchmarks, including CIFAR-10/100, MIRFLICKR-25K, and NUS-WIDE.

Visualization of semantic feature maps. Similar visual words (indicated by indices like #175) activate on consistent semantic regions across different images of the same class.

Accuracy and Efficiency

On CIFAR-10, E$^2$BoWs achieved a Mean Average Precision (mAP) of 0.909, surpassing existing deep hashing methods. More impressively, the model demonstrated extreme efficiency; on CIFAR-100, it required approximately 400x fewer operations than the BHC baseline while maintaining superior accuracy.

Semantic Sharing and Generalization

A key innovation is the model's ability to handle fine-grained categories through "semantic sharing." By leveraging the ImageNet tree structure, the triplet loss allows related categories (e.g., different breeds of dogs) to share visual words. This spatial and semantic preservation allows the model to generalize effectively to unseen datasets like NUS-WIDE, outperforming standard DCNN features.

Conclusion

E$^2$BoWs successfully bridges the gap between the semantic depth of DCNNs and the structural efficiency of BoW models. By enforcing sparsity and preserving spatial information, it provides a scalable solution for high-performance image retrieval. Future work may address the current dependency on supervised category labels and explore dynamic vocabulary scaling for even larger datasets.

Summary updated

Abstract

Traditional Bag-of-visual Words (BoWs) model is commonly generated with many steps including local feature extraction, codebook generation, and feature quantization, etc. Those steps are relatively independent with each other and are hard to be jointly optimized. Moreover, the dependency on hand-crafted local feature makes BoWs model not effective in conveying high-level semantics. These issues largely hinder the performance of BoWs model in large-scale image applications. To conquer these issues, we propose an End-to-End BoWs (E$^2$BoWs) model based on Deep Convolutional Neural Network (DCNN). Our model takes an image as input, then identifies and separates the semantic objects in it, and finally outputs the visual words with high semantic discriminative power. Specifically, our model firstly generates Semantic Feature Maps (SFMs) corresponding to different object categories through convolutional layers, then introduces Bag-of-Words Layers (BoWL) to generate visual words for each individual feature map. We also introduce a novel learning algorithm to reinforce the sparsity of the generated E$^2$BoWs model, which further ensures the time and memory efficiency. We evaluate the proposed E$^2$BoWs model on several image search datasets including CIFAR-10, CIFAR-100, MIRFLICKR-25K and NUS-WIDE. Experimental results show that our method achieves promising accuracy and efficiency compared with recent deep learning based retrieval works.
BibTeX
@misc{liu2017e,
  author = {Xiaobin Liu and Shiliang Zhang and Tiejun Huang and Qi Tian},
  title = {{E$^2$BoWs: An End-to-End Bag-of-Words Model via Deep Convolutional Neural Network}},
  year = {2017},
  eprint = {1709.05903},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.05903},
}
Figure