Multilevel Context Representation for Improving Object Recognition

Andreas Kölsch, Muhammad Zeshan Afzal, Marcus Liwicki

In the race to build deeper and more complex neural networks, we often treat Convolutional Neural Networks (CNNs) as a strict one-way street: data goes in at the bottom, and a single "feature vector" comes out at the top to be classified. But does the very last layer really hold all the information we need?

A 2017 paper titled "Multilevel Context Representation for Improving Object Recognition" suggests that by ignoring the layers immediately preceding the output, we are leaving accuracy on the table. By simply "short-circuiting" the network to include features from multiple high-level blocks, the authors achieved performance gains that usually require much more computationally expensive techniques.

TL;DR

The researchers propose "AlexNet++" and "GoogLeNet++," architectural tweaks that concatenate features from the last two high-level convolutional blocks instead of just the final one. This simple change improves ImageNet classification accuracy by 1-2% and allows a single-crop prediction to match the accuracy of a traditional 144-crop ensemble, making inference up to 144x faster.

What Problem Does This Paper Address?

Standard CNN architectures are hierarchical: early layers detect simple edges, while deeper layers detect complex, class-specific patterns (like a "dog face" or "car wheel"). Usually, only the output of the very last convolutional layer is fed into the final classifier.

The authors argue this is a missed opportunity. While the top layer is highly specialized, the layers just below it contain rich "contextual" information that provides better scale and translation invariance. In complex scenes—where objects might be partially occluded, appearing at odd angles, or vary significantly in size—the final layer alone might be too "brittle." The challenge was to find a way to integrate this multilevel context without significantly increasing the computational "tax" of the model.

Methods

The authors introduced a straightforward modification to two classic architectures:

  1. AlexNet++: In the original AlexNet, the fc6 layer only sees the output of conv5. In the "++" version, the researchers take the output of conv4, apply max-pooling to match dimensions, normalize the values using a tanh function, and concatenate them with the conv5 output. This combined "multilevel" vector is then fed into the classifier.
  2. GoogLeNet++: Following a similar logic, they added a connection from a slightly earlier Inception block (specifically inception4e) directly to the final concatenation layer before the classifier.

Crucially, these are "standalone" solutions. Unlike some methods that require pre-training or complex auxiliary losses, these networks can be trained from scratch.

Datasets

The models were trained and evaluated on the gold standard of the era: ImageNet (ILSVRC).

  • Training: 1.2 million labeled images across 1,000 classes.
  • Validation: 50,000 images used for benchmarking.
  • Preprocessing: Images were scaled to 256x256, mean-subtracted, and randomly cropped to the network's input size (227x227 for AlexNet, 224x224 for GoogLeNet).

Key Insights and Ideas

  • The "Aha" Moment on Efficiency: The most striking insight is the trade-off between architectural complexity and data augmentation. Usually, to squeeze out an extra 1% of accuracy, researchers use "multi-crop" testing (running the model on 144 different versions of the same image and averaging the results). The authors found that their "++" models achieved similar accuracy using just one center crop.
  • Normalization is Key: In their pre-study, the authors found that simply smashing two layers together doesn't work. Because different layers have different activation scales, the network fails to learn. Using L2-normalization or a tanh function to balance the contributions of the different layers was essential for the "multilevel" approach to succeed.
  • Context vs. Depth: While the industry trend was simply adding more layers (ResNet, PolyNet), this paper shows that how you connect existing high-level layers is just as important as how many you have.

Key Results

  • Accuracy Boost: Both AlexNet++ and GoogLeNet++ outperformed their original counterparts across the board. For example, GoogLeNet++ achieved a Top-1 accuracy of 69.42% (center crop) compared to the original's 68.89%.
  • Massive Speedup at Inference: Because AlexNet++ with one crop performs as well as AlexNet with 144 crops, it is effectively 144 times faster during the testing/deployment phase for the same level of accuracy.
  • Negligible Training Overhead: Adding these connections is computationally "cheap." GoogLeNet++ was only 0.4% slower to train than the original GoogLeNet, making it a very high-ROI architectural change.

Limitations and Future Work

The authors noted that while their method is effective, it didn't reach the absolute state-of-the-art numbers reported by others (like Szegedy et al.) because they intentionally used less aggressive data augmentation to keep their experiments controlled and reproducible.

For future work, they suggested applying this multilevel context idea to Recurrent Neural Networks (RNNs) and Convolutional-Recurrent Neural Networks (CRNNs), exploring whether temporal context can be improved in the same way spatial context was improved here.

Summary updated

Abstract

In this work, we propose the combined usage of low- and high-level blocks of convolutional neural networks (CNNs) for improving object recognition. While recent research focused on either propagating the context from all layers, e.g. ResNet, (including the very low-level layers) or having multiple loss layers (e.g. GoogLeNet), the importance of the features close to the higher layers is ignored. This paper postulates that the use of context closer to the high-level layers provides the scale and translation invariance and works better than using the top layer only. In particular, we extend AlexNet and GoogLeNet by additional connections in the top n layers. In order to demonstrate the effectiveness of the proposed approach, we evaluated it on the standard ImageNet task. The relative reduction of the classification error is around 1-2% without affecting the computational cost. Furthermore, we show that this approach is orthogonal to typical test data augmentation techniques, as recently introduced by Szegedy et al. (leading to a runtime reduction of 144 during test time).
BibTeX
@inproceedings{kolsch2017multilevel,
  author = {Andreas Kölsch and Muhammad Zeshan Afzal and Marcus Liwicki},
  title = {{Multilevel Context Representation for Improving Object Recognition}},
  booktitle = {WML@ICDAR},
  year = {2017},
  url = {https://openreview.net/forum?id=x9-w79N2Yep},
}

Loading related content...

Figure