Deep Tensor Encoding

Biswa Sengupta, E. Vasquez, Yu QianarXiv 2017 · arXiv

Deep Tensor Encoding: Preserving Structural Information in Image Retrieval

Abstract

In standard deep learning pipelines for image retrieval, the final feature maps are typically flattened into a single vector, a process that discards valuable structural and spatial information. This paper proposes treating these features as multi-dimensional tensors and applying multilinear factorization techniques—such as Multilinear Principal Component Analysis (mPCA) and Tensor Singular Value Decomposition (t-SVD)—to preserve data fidelity. Evaluated on the Describable Textures Dataset (DTD), these tensor-based encoding strategies significantly improve retrieval accuracy and computational efficiency, offering a robust solution for data-constrained domains like fashion.

Introduction

Deep Convolutional Neural Networks (dCNNs) have revolutionized feature extraction, yet they suffer from a "flattening" bottleneck. In most Content-Based Image Retrieval (CBIR) systems, the 3D feature tensor (height × width × depth) produced by the final convolutional layer is collapsed into a 1D vector. This transformation ignores the inherent multi-linear structure—the spatial relationships and cross-channel correlations—essential for fine-grained discrimination.

Workflow for a content-based fashion image retrieval system.

This issue is particularly acute in fashion retrieval, where the distinction between items often rests on subtle texture variations (e.g., "polka-dotted" vs. "chequered") rather than broad categorical differences. Furthermore, deep models often struggle when labeled training data is scarce. By imposing structural constraints through tensor representation, the learning process is effectively regularized, leading to better generalization from limited datasets.

Methodology

The authors investigated several encoding strategies to determine which best maintains the "fidelity" of the original image for retrieval:

  1. Fisher Encoding: Utilizes Gaussian Mixture Models (GMMs) to describe how local image features deviate from a global average.
  2. Sparse Coding (k-SVD): Represents images as sparse combinations of basis vectors from an over-complete dictionary.
  3. Tensor Factorization: The core contribution, utilizing multilinear tools:
  4. mPCA: Finds projection matrices that maximize variance across all tensor dimensions simultaneously.
  5. t-SVD: A tensor-based SVD using the "t-product" and Fast Fourier Transforms (FFT) for efficiency.
  6. Low-rank + Sparse Factorization: Decomposes tensors into a stable low-rank component and a sparse "detail" component.

Experimental Results

The methods were tested using the Describable Textures Dataset (DTD), which contains 5,640 images across 47 texture categories. Features were extracted from the last convolutional layer of a pre-trained VGG-M network. Performance was measured using Mean Average Precision (MAP) for the top-10 retrieved results.

Method Top-1 MAP Latency (ms)
Sparse Coding 0.62 35
Fisher Vector 0.56 12
mPCA Dictionary 0.53 5
t-SVD Dictionary 0.53 2188
Raw Tensor 0.41 -
Example of retrieval results for patterned fashion items, highlighting the importance of texture fidelity.

The results indicate that any form of structured encoding vastly outperforms "Raw Tensors." While Sparse Coding achieved the highest precision (0.62), its computational cost is significant. mPCA emerged as the most practical solution for real-time applications, being over 7x faster than Sparse Coding and 2x faster than Fisher Vectors while maintaining competitive accuracy.

Conclusion

Treating deep features as tensors rather than vectors provides a powerful regularization mechanism that preserves structural information. While Sparse Coding offers peak accuracy, mPCA provides an optimal balance of speed and precision for production environments. Future work will explore the Riemannian geometry of tensors and the integration of multi-scale information from earlier convolutional layers to further enhance retrieval performance.

Summary updated

BibTeX
@misc{sengupta2017deep,
  author = {Biswa Sengupta and E. Vasquez and Yu Qian},
  title = {{Deep Tensor Encoding}},
  year = {2017},
  eprint = {1703.06324},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1703.06324},
}
Figure