Invariant Tensor Feature Coding

Yusuke Mukuta, Tatsuya Harada

Abstract: We propose a novel feature coding method that exploits invariance. We consider the setting where the transformations that preserve the image contents compose a finite group of orthogonal matrices. This is the case in many image transformations, such as image rotations and image flipping. We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier using convex loss minimization. From this result, we propose a novel feature modeling for principal component analysis and k-means clustering, which are used for most feature coding methods, and global feature functions that explicitly consider the group action. Although the global feature functions are complex nonlinear functions in general, we can calculate the group action on this space easily by constructing the functions as the tensor product representations of basic representations, resulting in the explicit form of invariant feature functions. We demonstrate the effectiveness of our methods on several image datasets.

Introduction

In computer vision, feature coding is the critical process of summarizing local features—such as those from a convolutional neural network (CNN) layer—into a single global vector for classification. A persistent challenge in this domain is achieving invariance: ensuring that an image's representation remains consistent despite transformations like horizontal flips or 90-degree rotations.

Traditionally, models address this through "data augmentation," which involves training on millions of transformed image variants. While effective, this approach is computationally expensive and does not guarantee that the model structurally learns the underlying symmetries. This research introduces a mathematically rigorous framework using group representation theory to embed invariance directly into the feature coding process, creating models that are inherently robust to specific transformation groups.

Methods

The proposed framework leverages Group Representation Theory to treat image transformations as a finite group of orthogonal matrices. The process involves several key stages to map local features into a space where they are invariant to these transformations.

Conceptual pipeline of the invariant feature coding framework, showing the transformation from local features to an invariant feature space via group representations.

1. Invariant Modeling (PCA & k-means)

The authors redefine standard dimensionality reduction (PCA) and clustering (k-means) to ensure that subspaces and cluster centroids are consistent with group actions. In "Invariant k-means," for instance, if a centroid is defined for a feature, the framework ensures that transformed versions of that centroid also exist within the model's structure.

2. Tensor Product Representation

To handle complex nonlinear features, such as those used in Bilinear Pooling or VLAD, the features are represented as tensor products. This allows the system to calculate how a transformation (like a rotation) affects a complex feature by observing its effect on simpler base features.

3. Projection to the Trivial Representation

A projection operator ($P_1$) is employed to extract the "trivial" component of the feature—the part that remains unchanged regardless of the applied group transformation. This results in several invariant coding variants, including Inv BP (Invariant Bilinear Pooling), Inv VLAD, and Inv VLAT.

Key Results and Insights

The effectiveness of these methods was evaluated across several benchmarks, including the Flickr Material Dataset (FMD), Describable Texture Dataset (DTD), and fine-grained recognition tasks like CUB-200-2011 (Birds) and Stanford Cars.

  • Efficiency and Accuracy: On the FMD dataset, the "Inv iBP" (Invariant improved Bilinear Pooling) method achieved 83.46% accuracy using only 82k dimensions. In contrast, standard iBP achieved 81.38% but required a much larger 525k-dimensional vector. This suggests that a significant portion of high-dimensional features in standard models is redundant noise related to orientation.
  • Robustness to Transformation: Standard models often suffer significant performance degradation when test images are randomly rotated or flipped. For example, standard Bilinear Pooling on the CUB dataset dropped from 77% to 37% accuracy under transformation, while the invariant models remained stable at approximately 81%.
  • Structural Invariance: The method is mathematically equivalent to training on infinite augmented data for the chosen group. However, it is far more efficient because the invariance is a structural property of the feature space rather than a learned behavior.

Conclusion

By treating invariance as a form of complexity reduction, this framework reduces the model's search space, leading to better generalization and reduced overfitting. While currently focused on finite groups and orthogonal transformations, the research provides a foundation for extending these principles to more complex affine warps and non-rigid transformations. The results demonstrate that embedding mathematical symmetry into feature coding can yield more compact, robust, and accurate computer vision models.

Summary updated

Abstract

We propose a novel feature coding method that exploits invariance. We consider the setting where the transformations that preserve the image contents compose a finite group of orthogonal matrices. This is the case in many image transformations, such as image rotations and image flipping. We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier using convex loss minimization. From this result, we propose a novel feature modeling for principal component analysis and k-means clustering, which are used for most feature coding methods, and global feature functions that explicitly consider the group action. Although the global feature functions are complex nonlinear functions in general, we can calculate the group action on this space easily by constructing the functions as the tensor product representations of basic representations, resulting in the explicit form of invariant feature functions. We demonstrate the effectiveness of our methods on several image datasets.
BibTeX
@misc{mukuta2019invariant,
  author = {Yusuke Mukuta and Tatsuya Harada},
  title = {{Invariant Tensor Feature Coding}},
  year = {2019},
  eprint = {1906.01857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.01857},
}
Figure