Are You Sure You Want To Do That? Classification with Verification

Harris Chan, Atef Chaudhury, Kevin ShenarXiv 2018 · arXiv

Abstract

Standard supervised classification models typically rely on internal "memorization" of class features, a strategy that scales poorly as the number of classes increases. This paper introduces the Recognition-Verification Neural Network (RVNN), a model that classifies images by iteratively querying external reference images. By adopting a "hypothesize-and-verify" approach, the RVNN achieves state-of-the-art accuracy on the MNIST dataset with approximately 50% fewer parameters than standard baselines. However, the study highlights that successful training requires a careful decomposition of recognition and verification tasks to prevent the model from defaulting to standard recognition-heavy behavior.

Introduction

Most modern neural networks are recognition-heavy. To identify an object, a model must implicitly store the features of every possible class it might encounter. While effective for simple tasks, this approach becomes inefficient when dealing with thousands of classes or limited training data.

Humans often solve this problem by forming a hypothesis and then verifying it against external information—for example, looking up a reference photo to confirm a rare dog breed. The RVNN explores whether neural networks can adopt this behavior to become more parameter-efficient and robust.

Conceptual comparison of standard recognition, verification, and the proposed RVNN architecture.

The RVNN Architecture

The RVNN architecture is designed to offload internal memory requirements by performing multiple steps of computation and looking at reference images. It consists of three primary components:

  1. CNN Module ($f_{cnn}$): This module extracts features from both the input image and a "support" (reference) image.
  2. Recurrent Module ($f_{rnn}$): A Gated Recurrent Unit (GRU) tracks the history of hypotheses and determines the next step in the query process.
  3. Query Component ($f_q$): This component converts the RNN's hidden state into a discrete query for a specific class's reference image.
The iterative architecture of the RVNN, showing the flow from CNN features to RNN-based query proposals over time steps.

Because the query process is discrete and non-differentiable, the authors utilize the Gumbel-Softmax trick and Straight-Through estimators. These techniques allow the model to learn a search policy by backpropagating through the decision of which image to retrieve.

Key Insights and Challenges

Parameter Efficiency vs. The "Lazy" Model

The RVNN demonstrated significant parameter efficiency, matching baseline accuracy while using only 13.9k parameters compared to the baseline's 27.7k. However, researchers encountered a "lazy model" problem: in unified architectures, the model often ignored reference images, using the RNN as extra processing power for standard recognition rather than a tool for verification.

The Importance of Decomposition

The study found that an intelligent query policy only emerged when recognition and verification components were explicitly separated. When forced to use a pre-trained comparator, the RNN learned to query the most likely classes first, outperforming random search. This suggests that a "recognition-then-verification" pipeline—where recognition narrows down candidates for a dedicated verification step—is more effective than a simultaneous approach.

Experimental Results

Evaluated on the MNIST benchmark, the RVNN reached 99.09% accuracy with half the parameters of the baseline. To test the model's reliance on external queries, researchers replaced reference images with incorrect ones during inference. While accuracy dropped (from 99.29% to 97.44%), the relatively small decrease confirmed that unified models still lean heavily on internal recognition. Furthermore, the RVNN was found to be less sample-efficient than baselines when training data was scarce, indicating that learning an effective query policy itself requires substantial data.

Conclusion and Future Work

The RVNN demonstrates that iterative verification can significantly reduce model size without sacrificing accuracy. While tested on MNIST, the benefits of this approach are expected to be more pronounced in large-scale datasets with thousands of classes, such as ImageNet. Future research will focus on applying this recognition-then-verification pattern to few-shot learning, where models must identify new objects from very few examples.

Summary updated

BibTeX
@misc{chan2018are,
  author = {Harris Chan and Atef Chaudhury and Kevin Shen},
  title = {{Are You Sure You Want To Do That? Classification with Verification}},
  year = {2018},
  eprint = {1809.02652},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.02652},
}
Figure