Convolutional Neural Networks Applied to House Numbers Digit Classification

Pierre Sermanet, Soumith Chintala, Yann LeCun

Abstract

We classify digits of real-world house numbers using convolutional neural networks (ConvNets). Unlike many popular vision approaches that are hand-designed, ConvNets can automatically learn a unique set of features optimized for a given task. We augmented the traditional ConvNet architecture by learning multi-stage features and by using Lp pooling, establishing a new state-of-the-art of 94.85% accuracy on the SVHN dataset (45.2% error improvement).

Introduction

In 2012, the transition from hand-crafted computer vision features to fully learned deep learning systems marked a pivotal shift in the field. This research from the NYU Courant Institute demonstrated that Convolutional Neural Networks (ConvNets) could master the challenges of natural scene digit recognition. By applying specific architectural enhancements to the Street View House Numbers (SVHN) dataset, the authors proved that learned features could significantly outperform traditional methods like HOG or template matching.

The SVHN Dataset

The Street View House Numbers (SVHN) dataset represents a significant step up in difficulty from the classic MNIST benchmark. It contains over 600,000 labeled digits extracted from Google Street View images. These 32x32 pixel cropped color images present real-world challenges including low resolution, motion blur, varying illumination, and cluttered backgrounds.

Methodology

The authors utilized a 2-stage ConvNet architecture with two primary innovations: Multi-Stage Features and Lp Pooling.

Multi-Stage Features (MS)

In a standard ConvNet, only the final layer's output is typically fed into the classifier. This paper introduces a "multi-stage" approach where the output of the first stage is branched out, subsampled, and concatenated with the second stage's output. This ensures the classifier receives both high-level abstract shapes and fine-grained local textures.

The multi-stage ConvNet architecture branches features from the first stage directly to the classifier.

Generalized Lp Pooling

Standard networks typically use Max or Average pooling. This work utilizes Lp pooling, defined as $O = (\sum I^p \times G)^{1/p}$. This generalization allows the network to tune the pooling behavior:

  • $p=1$ approximates Gaussian averaging.
  • $p=\infty$ approximates Max-pooling.

By using intermediate values, the network balances the preservation of strong features with noise suppression.

Results and Analysis

The proposed model achieved a state-of-the-art accuracy of 94.85%, representing a 45.2% improvement in error rate over previous non-ConvNet approaches. This brought machine performance significantly closer to the estimated human accuracy of 98% in unconstrained environments.

Validation error as a function of the p parameter in Lp pooling, showing optimal performance between p=2 and p=12.

A key insight was the performance of "in-between" pooling. As shown in the validation error analysis, neither $p=1$ (average) nor very high $p$ (max) was optimal. Instead, values between 2 and 12 provided the best results, suggesting that a weighted focus on strong signals is superior to simple averaging or extreme max-pooling.

Conclusion

This work highlights the power of end-to-end feature learning. While multi-stage features provided varying gains depending on the task—offering more significant improvements for complex textures like pedestrian detection than for house numbers—the combination of architectural flexibility and large-scale data proved decisive. Future work suggests that artificial scale deformations and unsupervised pre-training could further bridge the gap to human-level perception.

Summary updated

Abstract

We classify digits of real-world house numbers using convolutional neural networks (ConvNets). ConvNets are hierarchical feature learning neural networks whose structure is biologically inspired. Unlike many popular vision approaches that are hand-designed, ConvNets can automatically learn a unique set of features optimized for a given task. We augmented the traditional ConvNet architecture by learning multi-stage features and by using Lp pooling and establish a new state-of-the-art of 94.85% accuracy on the SVHN dataset (45.2% error improvement). Furthermore, we analyze the benefits of different pooling methods and multi-stage features in ConvNets. The source code and a tutorial are available at eblearn.sf.net.
BibTeX
@inproceedings{sermanet2012convolutional,
  author = {Pierre Sermanet and Soumith Chintala and Yann LeCun},
  title = {{Convolutional Neural Networks Applied to House Numbers Digit Classification}},
  booktitle = {ICPR},
  year = {2012},
  url = {https://openreview.net/forum?id=sKwlXCWR5Z},
}
Figure