Abstract
One-shot image classification remains a significant challenge due to the "noisy prototype" problem, where a single, unrepresentative training image fails to capture the true class distribution. This paper introduces RestoreNet, a class-agnostic transformation model that learns to map outlier features toward their true class centers. By employing a simple Multi-Layer Perceptron (MLP) with a residual connection and a hard-sample mining strategy, RestoreNet achieves state-of-the-art results on several benchmarks, including miniImageNet and CUB-200.
Introduction
Most modern image classifiers are "data hungry," requiring thousands of images to learn a category. Metric learning—specifically Prototypical Networks—has emerged as a popular solution for low-data regimes. These models map images into a feature space where images of the same class cluster together around a "prototype" (the mean vector of that class).
The "One-Shot" problem occurs when only one training image is available. In this case, that single image is the prototype. If that image is an outlier—perhaps showing an object from an unusual angle or with a distracting background—the prototype will be far from the "true" center of the class. This shifts the decision boundary, causing the model to misclassify new images. The authors identify this as the noisy prototype problem.
Methods: The RestoreNet Architecture
The authors propose RestoreNet, a class-agnostic transformation network designed to move a "noisy" single-image feature toward the "true" class center.
RestoreNet is implemented as a small Multi-Layer Perceptron (MLP) that takes the feature vector of a single image and outputs a "restored" version. A key feature is the skip-connection: during inference, the final prototype is the average of the original (noisy) feature and the restored feature. This residual-like structure prevents the model from "over-correcting" and losing the original identity of the image.
The authors also introduce Self-RestoreNet, which incorporates unlabeled "query" images to further refine the prototype through a self-training loop, as shown in the figure above.
Training on Outliers
To train the MLP, the authors use "base classes" with abundant data. For each base class, they calculate the true center (the mean of all images) and intentionally select the images farthest from that center to act as "noisy" examples. The MLP is trained to map these outliers back toward the true center using a squared Euclidean distance loss. This "hard sample mining" approach teaches the network exactly how to handle the most difficult outliers.
Experimental Results
The model was evaluated on four diverse benchmarks: miniImageNet, CIFAR-100, Caltech-256, and CUB-200 (a fine-grained bird dataset).
RestoreNet achieved state-of-the-art performance, reaching 59.28% accuracy on the 5-way, 1-shot miniImageNet task. When combined with self-training, performance improved to 61.14%. The model provided a consistent boost of 1–2% across all four datasets compared to strong baselines.
As shown in the t-SNE visualization, the initial one-shot prototypes (purple triangles) are visibly pushed toward the true class centers (red stars) after applying RestoreNet (green triangles). This confirms that the class-agnostic restoration effectively mitigates the noise caused by pose, lighting, or background.
Discussion and Conclusion
A primary insight of this work is that feature-space "noise" is somewhat universal; a model can learn to "fix" a feature vector without needing to know the specific object category. Furthermore, RestoreNet is computationally efficient; unlike complex meta-learning approaches like MAML, it is a simple regression task that can be trained in seconds.
However, the model exhibits sensitivity to the hyperparameter $\lambda$ (the number of noisy images used during training), which varies by dataset. Future work may involve integrating RestoreNet into more complex meta-learning frameworks or extending it to tasks like few-shot object detection.
This paper was presented at AAAI 2020.
Summary updated