Multilevel Context Representation for Improving Object Recognition

Andreas Kölsch, Muhammad Zeshan Afzal, Marcus Liwicki

Standard Convolutional Neural Networks (CNNs) are typically designed as a linear hierarchy where information flows from low-level edge detectors to high-level semantic features. In this paradigm, only the final feature vector is used for classification. However, research suggests that the layers immediately preceding the output contain rich contextual information that is often discarded. This paper proposes a "multilevel context" approach that integrates features from multiple high-level blocks to improve scale and translation invariance.

The Multilevel Context Approach

The core hypothesis is that features close to the higher layers provide better invariance to transformations than the top layer alone. By "short-circuiting" the network to include outputs from the penultimate layers, the model gains a more robust representation of the input.

Proposed architecture with skip connections from earlier blocks to the classifier.

To implement this, the authors introduced modifications to two standard architectures:

  • AlexNet++: Instead of feeding only conv5 into the first fully connected layer (fc6), the output of conv4 is also included. The conv4 features are max-pooled to match spatial dimensions and normalized using a tanh function to ensure both layers contribute equally to the final representation.
  • GoogLeNet++: A similar connection is added from an earlier Inception block (inception4e) directly to the final concatenation layer before the classifier.

Addressing Fine-Grained Recognition

One of the primary motivations for using multilevel context is the difficulty of distinguishing between highly similar classes. In ImageNet, for instance, the difference between a "Tabby cat," a "Tiger cat," and an "Egyptian cat" can be subtle.

Fine-grained classification challenges in ImageNet, where subtle contextual cues are vital.

The authors argue that while the top layer might focus on specific discriminative parts, the preceding layers provide the broader context necessary to handle variations in scale, occlusion, and orientation that are common in these complex categories.

Experimental Results and Efficiency

The proposed models were evaluated on the ILSVRC ImageNet dataset. The results demonstrated a consistent improvement in classification accuracy:

  • Accuracy Gains: Both AlexNet++ and GoogLeNet++ reduced classification error by approximately 1-2% compared to their baseline counterparts. GoogLeNet++ achieved a Top-1 accuracy of 69.42% using a single center crop, outperforming the original's 68.89%.
  • Inference Speedup: A significant finding was that the "++" models with a single-crop input could match the performance of standard models using a 144-crop ensemble. This effectively provides a 144x speedup during inference for the same level of accuracy.
  • Computational Cost: The architectural changes are remarkably efficient. Training GoogLeNet++ was only 0.4% slower than the original, making it a low-cost upgrade for significant performance gains.

Conclusion

By rethinking the connectivity of high-level layers, this work demonstrates that multilevel context is a powerful tool for improving object recognition. The approach is orthogonal to traditional data augmentation techniques and provides a path toward highly efficient, high-accuracy inference. Future work may extend these principles to temporal context in recurrent architectures.

Summary updated

Abstract

In this work, we propose the combined usage of low- and high-level blocks of convolutional neural networks (CNNs) for improving object recognition. While recent research focused on either propagating the context from all layers, e.g. ResNet, (including the very low-level layers) or having multiple loss layers (e.g. GoogLeNet), the importance of the features close to the higher layers is ignored. This paper postulates that the use of context closer to the high-level layers provides the scale and translation invariance and works better than using the top layer only. In particular, we extend AlexNet and GoogLeNet by additional connections in the top $n$ layers. In order to demonstrate the effectiveness of the proposed approach, we evaluated it on the standard ImageNet task. The relative reduction of the classification error is around 1-2% without affecting the computational cost. Furthermore, we show that this approach is orthogonal to typical test data augmentation techniques, as recently introduced by Szegedy et al. (leading to a runtime reduction of 144 during test time).
BibTeX
@inproceedings{kolsch2017multilevel,
  author = {Andreas Kölsch and Muhammad Zeshan Afzal and Marcus Liwicki},
  title = {{Multilevel Context Representation for Improving Object Recognition}},
  booktitle = {WML@ICDAR},
  year = {2017},
  url = {https://openreview.net/forum?id=x9-w79N2Yep},
}
Figure