Improving Accuracy of Nonparametric Transfer Learning via Vector Segmentation

Vincent Gripon, Ghouthi B. Hacene, Matthias Löwe, Franck Vermet

Abstract

Nonparametric transfer learning, while robust for incremental tasks, often suffers from the "curse of dimensionality" when using high-dimensional feature vectors from pre-trained deep neural networks. This paper proposes $c$-segmentation, a method that divides feature vectors into smaller subvectors for independent search and majority voting. We demonstrate that this technique significantly improves classification accuracy across vision and audio datasets by isolating informative signals from irrelevant noise, all without increasing computational costs.

Introduction

Transfer learning has become the standard for solving niche AI problems. Instead of training a massive model from scratch, practitioners use a pre-trained Deep Neural Network (DNN) to extract "features" (high-dimensional vectors) and then apply a simple classifier like $k$-Nearest Neighbors ($k$-NN) to categorize them.

However, DNNs are typically trained on massive, general datasets like ImageNet. When applied to specific, different tasks, a large portion of the feature vector may be irrelevant noise. In high-dimensional spaces, this noise accumulates, causing standard Euclidean distance metrics to fail. The authors investigate whether mitigating this "curse of dimensionality" is possible by breaking the vector apart.

The $c$-Segmentation Method

The authors propose a technique called $c$-segmentation, which follows a four-step process:

  1. Splitting: A feature vector of dimension $d$ is split into $c$ equal-sized segments (subvectors) of dimension $d/c$.
  2. Independent Search: For a new test sample, a $k$-NN search is performed independently within each of the $c$ subspaces using a dictionary of known examples.
  3. Voting: Each subspace "votes" for a class based on its local nearest neighbor.
  4. Aggregation: The final classification is determined by a majority vote across all $c$ segments.

This approach is mathematically grounded in large deviation theory, suggesting that if information is "sparse" (concentrated in a few coordinates), segmentation allows the "signal" to outvote the "noise."

Experimental Setup

The researchers evaluated their method using three different DNN architectures across four datasets:

  • Feature Extractors: Inception V3 (2,048 dimensions), SqueezeNet (1,000 dimensions), and an AudioSet-trained model (1,280 dimensions).
  • Vision Datasets: CIFAR10 (low-resolution 32x32 images) and two subsets of ImageNet (ImageNet1 & ImageNet2) consisting of 10 classes each, specifically excluding classes used during initial DNN training.
  • Audio Dataset: A subset of 10 classes from AudioSet (e.g., "cat," "fireworks," "snoring").

Key Results and Discussion

The experiments consistently showed that an intermediate number of segments ($c > 1$) yielded the best results. In almost every scenario, the "naive" approach of using the full vector ($c=1$) was outperformed by a segmented approach.

Vision Domain Performance

Using Inception V3 on CIFAR10, accuracy improved from a baseline of 85.19% ($c=1$) to 87.81% at $c=16$. Similar gains were seen across ImageNet subsets, where peak performance was typically reached at $c=16$ or $c=64$.

Classification accuracy for Inception V3 (1-NN) across different numbers of segments $c$.

Audio Domain Performance

The impact was even more pronounced in the audio domain. For the AudioSet features, the best performance was found at $c=40$, where 1-NN accuracy rose from 60.5% to 72.4%.

Classification accuracy for AudioSet features (1-NN and 5-NN) as a function of segment count $c$.

Robustness and Efficiency

  • Robustness through Voting: By segmenting the vector, informative coordinates are isolated. While noisy segments might return random results, informative segments consistently vote for the correct class, allowing the signal to win the majority vote.
  • Zero Computational Cost: Calculating the distance of one 1,000-D vector is computationally equivalent to calculating the distances of ten 100-D vectors, meaning this accuracy boost comes for free in terms of operations and memory.
  • Optimal $c$: If $c$ becomes too large (e.g., $c=256$), accuracy begins to drop as segments become too small to contain meaningful patterns.

Conclusion and Future Work

Vector segmentation provides a simple yet powerful way to improve the accuracy of distance-based transfer learning. By treating a single high-dimensional search as a committee of lower-dimensional searches, we can effectively filter out noise inherent in general-purpose feature extractors. Future work will explore whether this segmentation provides similar boosts to other downstream classifiers like Support Vector Machines (SVMs) and investigate methods to automatically determine the optimal segment count $c$ based on dataset properties.

Summary updated

BibTeX
@inproceedings{gripon2018improving,
  author = {Vincent Gripon and Ghouthi B. Hacene and Matthias Löwe and Franck Vermet},
  title = {{Improving Accuracy of Nonparametric Transfer Learning via Vector Segmentation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  doi = {10.1109/icassp.2018.8462273},
}
Figure