Visual recognition in the wild by sampling deep similarity functions

Mikhail Usvyatsov, Konrad Schindler

Abstract

Standard deep learning classifiers often struggle with "domain shift," where test environments differ significantly from training data. This paper proposes a shift from direct classification to similarity learning using Siamese networks. By training a model to learn similarity functions rather than class labels, robots can achieve better generalization. Our results show that comparing new images to a few random training examples reduces classification errors by up to 64% across diverse datasets, including a novel robot-collected door dataset.

Introduction

Deep learning typically assumes a "stable domain," where training data encompasses all possible variations encountered at test time. However, for robots operating "in the wild," this assumption is frequently violated. A robot trained to recognize open doors in one facility may fail in another due to changes in lighting, wall color, or door materials.

The core issue is that direct CNN classifiers are prone to overfitting to spurious correlations. If a training set for "open doors" consistently features a specific background color, the network may incorrectly associate that color with the "open" state. This phenomenon, known as domain shift, causes the network to ignore robust discriminative features in favor of environment-specific details.

Methods

To address these challenges, we propose a methodology centered on similarity learning rather than direct label prediction.

Siamese Architecture

We utilize a Siamese network architecture, typically based on a VGG16 backbone, featuring two identical branches with shared weights. Unlike standard classifiers that output class probabilities for a single image, this network processes a pair of images and predicts a similarity score $d_c$ ranging from 0 to 1.

Siamese architecture for similarity learning.

Learning the Embedding

By training on image pairs, the network learns a high-dimensional embedding space where images of the same class are clustered together and different classes are separated. This approach forces the model to identify what makes two objects "the same," focusing on intrinsic features rather than memorizing the appearance of specific classes.

Classification by Sampling

At test time, a query image $Q$ is classified by comparing it to a small set of $k$ random exemplars from the training set for each class. The similarity scores are averaged, and the class with the highest average similarity is selected. This "sampling" approach is computationally efficient and avoids the need for exhaustive comparisons.

Datasets and Evaluation

The proposed method was evaluated across four distinct datasets to demonstrate its robustness:

  1. Open and Closed Doors: A new RGB-D dataset collected by a mobile robot (shown below) across two different warehouses. This dataset specifically tests domain shift by training in one location and testing in another.
  2. NYU2: RGB-D indoor scenes used for distinguishing between "office" and "home office" environments.
  3. Pasadena Street Trees: RGB images of tree species from Google Streetview, characterized by high background variability.
  4. Omniglot: Handwritten characters used for testing generalization across writers (INST) and to entirely new symbols (SYMB).
Mobile robot platform and the door classification task.

Key Insights

  • Global Embedding Properties: Siamese networks learn an approximately global embedding, preserving distance relationships even across arbitrary data subsets.
  • Robustness of Random Sampling: High performance is achieved even with a small number of random exemplars ($k=10$), suggesting that "perfect" representative images are not required.
  • Mitigating Spurious Features: Pair-based learning naturally filters out accidental correlations (like background color) because such features are unlikely to be consistent across randomly sampled pairs.

Results

The similarity-based approach consistently outperformed standard classification baselines:

  • Error Reduction: Classification errors were reduced by 28% to 64% across all tested datasets.
  • Robot Door Task: Accuracy on the challenging door dataset improved from a baseline of 86.1% to over 91%.
  • Omniglot SYMB: In the most difficult scenario—recognizing symbols never seen during training—the Siamese approach achieved ~81.5% accuracy, significantly higher than the ~72% achieved by the baseline.
  • Comparative Performance: Our method outperformed specialized few-shot learning algorithms like "Learning to Compare" (LtC), even when those algorithms were provided with more exemplars.

Conclusion

Moving from direct classification to similarity learning offers a powerful way to handle domain shift in robotic vision. By focusing on relative similarity rather than absolute labels, models become more robust to the "intra-class variation" found in real-world environments. Future work will explore "learning the consensus"—using secondary networks to optimize how similarity scores from multiple exemplars are combined.

Summary updated

Abstract

Recognising relevant objects or object states in its environment is a basic capability for an autonomous robot. The dominant approach to object recognition in images and range images is classification by supervised machine learning, nowadays mostly with deep convolutional neural networks (CNNs). This works well for target classes whose variability can be completely covered with training examples. However, a robot moving in the wild, i.e., in an environment that is not known at the time the recognition system is trained, will often face \emph{domain shift}: the training data cannot be assumed to exhaustively cover all the within-class variability that will be encountered in the test data. In that situation, learning is in principle possible, since the training set does capture the defining properties, respectively dissimilarities, of the target classes. But directly training a CNN to predict class probabilities is prone to overfitting to irrelevant correlations between the class labels and the specific subset of the target class that is represented in the training set. We explore the idea to instead learn a Siamese CNN that acts as similarity function between pairs of training examples. Class predictions are then obtained by measuring the similarities between a new test instance and the training samples. We show that the CNN embedding correctly recovers the relative similarities to arbitrary class exemplars in the training set. And that therefore few, randomly picked training exemplars are sufficient to achieve good predictions, making the procedure efficient.
BibTeX
@inproceedings{usvyatsov2019visual,
  author = {Mikhail Usvyatsov and Konrad Schindler},
  title = {{Visual recognition in the wild by sampling deep similarity functions}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  doi = {10.1109/icra.2019.8794162},
}
Figure