Enhancing ConvNeXt for efficient small-size image classification

Jianwei Feng, J. Mo, Hengliang Tan, Shuo YangResearch Square 2025 · DOI

TL;DR

This paper introduces an enhanced version of the ConvNeXt architecture specifically optimized for low-resolution, small-scale image datasets. By integrating a symmetrical inverted bottleneck, a "twice-patch" embedding strategy, and a novel spatial coordinate attention mechanism, the authors achieve state-of-the-art accuracy on benchmarks like CIFAR and Tiny-ImageNet while significantly reducing parameter counts.

What Problem Does This Paper Address?

While Vision Transformers (ViTs) and modern CNNs like ConvNeXt have revolutionized image classification, they are primarily designed for high-resolution images (typically 224x224 or larger). When these "heavyweight" models are applied to small-size datasets with low-resolution images (like the 32x32 pixels of CIFAR-10), they often suffer from lower accuracy and poor computational efficiency.

The authors identify that standard ConvNeXt's large kernel sizes (7x7) and aggressive downsampling ("patchify") layers destroy critical spatial information in small images. Furthermore, these models often experience "feature break"—a phenomenon where many neurons become inactive or saturated, leading to redundant feature maps.

Methods

The authors propose several structural modifications to transform ConvNeXt into a "small-data" specialist:

  1. Small Kernel Sizes: They replaced the standard 7x7 depthwise convolutions with 3x3 kernels. In low-res images, a 7x7 kernel covers too much of the image at once, losing fine-grained context.
  2. Symmetrical Inverted Bottleneck: Inspired by MobileNetV2 and Sandglass blocks, they moved depthwise convolutions to both the start and end of the residual path, creating a more balanced flow for feature extraction.
  3. Twice-Patch Embedding: Instead of a single aggressive downsampling step, they use a dynamic methodology that generates adaptive patch sizes based on input dimensions, ensuring the first layer always processes a consistent 32x32 feature map.
  4. Hybrid Normalization: They innovatively combined Batch Normalization (BN) at the block level (to handle inter-sample relationships) and Layer Normalization (LN) at the stage level (to stabilize feature relationships within a single sample).
  5. Global Response Normalization (GRN): To combat "feature break," they added a GRN layer that uses L2-norm aggregation to create competition between channels, encouraging the model to learn more diverse and distinct features.
  6. Spatial Coordinate Attention (SACA): This is the "secret sauce." It combines Coordinate Attention (which decomposes global pooling into 1D encodings to preserve positional data) with Spatial Attention (which focuses on "where" the important parts of the image are).

Datasets

The model was evaluated on four widely used small-size image benchmarks:

  • CIFAR-10: 60,000 32x32 color images across 10 classes.
  • CIFAR-100: Similar to CIFAR-10 but with 100 classes, presenting a higher complexity.
  • Tiny-ImageNet: A subset of ImageNet with 200 classes and 64x64 resolution.
  • Fashion-MNIST: 70,000 28x28 grayscale images of clothing items.

Key Insights and Ideas

  • Bigger isn't always better: While the trend in CV is toward larger kernels and higher resolutions, this paper proves that "downsizing" the architecture's receptive field is essential for low-res data.
  • Fixing Feature Collapse: The visualization of feature maps (via Grad-CAM) showed that standard ConvNeXt often has "dead" channels when working with small images. The inclusion of GRN effectively "wakes up" these neurons, leading to a much richer representation of the object.
  • The Power of Coordinate Attention: By splitting global pooling into horizontal and vertical components, the model can locate objects more precisely than standard global average pooling, which is crucial when the total pixel count is already very low.

Key Results

The proposed model (Base version) achieves impressive results with only 7.3 million parameters:

  • CIFAR-10: 98.14% accuracy, outperforming much larger models like ViT-S (48.1M params) and Swin-T (27.5M params).
  • CIFAR-100: 85.16% accuracy, a significant jump over the standard ConvNeXt-T (67.58%).
  • Tiny-ImageNet: 73.23% accuracy, setting a high bar for efficiency vs. accuracy (see the "bubble" chart in Figure 6 of the paper).
  • Fashion-MNIST: 96.32% accuracy, surpassing specialized models like DenseNet-BC.

Essentially, the model "punches above its weight," delivering performance comparable to models 4-6 times its size.

Limitations and Future Work

The authors note that on very simple datasets with few categories (like CIFAR-10), their complex architecture can occasionally lead to slight overfitting compared to the absolute latest iterations of Conv2NeXt, though it remains superior on more complex tasks like CIFAR-100.

Future work could involve exploring how these small-size optimizations perform in real-time edge computing scenarios (like mobile devices or IoT sensors) where both memory and resolution are strictly limited. The authors also suggest further investigating the synergy between different attention mechanisms in even deeper architectures.

Summary updated

Abstract

<title>Abstract</title> Vision Transformers (ViTs) have achieved remarkable success in computer vision, particularly with the advent of the Swin Transformer (Swin-T). Recently, ConvNeXt was proposed to revisit the architecture of convolutional neural networks by incorporating techniques from Swin-T, which achieves competitive performance. However, ConvNeXt exhibited relatively lower accuracy and efficiency on small-size datasets with low-resolution images. To address this issue, based on the structure of ConvNeXt, we propose to employ small kernel sizes to better capture features from low-resolution images. A symmetrical inverted bottleneck structure inspired by MobileNet is used to refine the traditional residual block design. We combine the batch normalization and layer normalization to strengthen the relationships between different samples. We use a twice-patch embedding methodology to dynamically generate adaptive patch sizes according to different input image sizes. Subsequently, we integrate the global response normalization to amplify the contrast and specificity of multiple channels for learning diverse features. Finally, we introduce a novel spatial coordinate attention mechanism that effectively captures global feature information. The proposed method demonstrates superior performance on small-size datasets of CIFAR-10, CIFAR-100, Tiny-ImageNet and Fashion-MNIST with fewer parameters, which confirms the effectiveness and efficiency of our approach.
BibTeX
@inproceedings{feng2025enhancing,
  author = {Jianwei Feng and J. Mo and Hengliang Tan and Shuo Yang},
  title = {{Enhancing ConvNeXt for efficient small-size image classification}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-7965188/v1},
}
Figure