Revisiting non-learned operators based deep learning for image classification: a lightweight directional-aware network

Yuwei Guo, Wenhao Zhang, Yupeng Gao, Licheng Jiao, Shuo Wang, Jiabo Du, Fang Liu

Introduction

Modern computer vision is dominated by increasingly complex deep neural networks. While these models achieve state-of-the-art performance, they often require massive datasets and significant computational resources, acting as "black boxes" with limited interpretability. To address these challenges, researchers have explored hybrid architectures that combine fixed, "non-learned" mathematical operators with traditional neural layers. These operators, such as Wavelet or Fourier transforms, provide a stable feature extraction foundation, theoretically improving efficiency and robustness.

However, traditional non-learned hybrids have struggled to keep pace with purely learned models. This paper introduces the Lightweight Directional-Aware Network (LDAN), a novel architecture that revitalizes the use of non-learned operators by replacing standard wavelets with Shearlets and introducing a directional-attention mechanism.

The Geometric Challenge: Why Wavelets Fall Short

The primary limitation of previous hybrid models lies in the mathematical nature of the Wavelet transform. Wavelets are isotropic, meaning their support bases are essentially square. While effective for capturing horizontal and vertical edges, they struggle with smooth curves and diagonal features common in natural images.

Comparison of (a) Wavelet and (b) Shearlet tiling for approximating a curved edge.

As shown in the figure above, wavelets (a) require a large number of small squares to approximate a curve, leading to poor sparsity and high noise. In contrast, Shearlets (b) utilize elongated, "sheared" rectangular bases that can align more precisely with the geometry of the image, resulting in a much more efficient and sparse representation.

The LDAN Architecture

LDAN is designed to be both lightweight and directionally sensitive. It consists of three core modules:

1. ShearletNet

At the heart of the network is the Shearlet filter bank. Unlike learned convolutional layers, these filters are mathematically predefined to cover multiple scales and orientations. This "baked-in" knowledge allows the network to capture complex geometric features without needing to learn them from scratch, significantly reducing the number of trainable parameters.

2. Directional-Aware (DA) Module

To further refine the features extracted by ShearletNet, the authors introduce the DA Module, which contains two sub-components:

  • Directional Attention (DA) Block: Not all directional features are equally important for classification. This block uses Global Average Pooling (GAP) and fully connected layers to learn a weight vector that dynamically emphasizes the most discriminative orientations.
  • Hierarchical Upsampling (HU) Block: To integrate multi-scale information, the HU block uses cascaded 3x3 convolutions with shared parameters. This approach expands the receptive field while keeping the parameter count low.

3. Pointwise Convolution Module

Finally, a 1x1 pointwise convolution layer integrates the multi-directional and multi-scale features. This serves as an efficient bridge to the final classification head, inspired by the efficiency of depthwise separable convolutions.

Experimental Results and Insights

The authors evaluated LDAN across several benchmarks, including CIFAR10, CIFAR100, STL10, and Tiny ImageNet. The results highlight three major advantages:

Efficiency and Accuracy

LDAN achieves a remarkable balance between size and performance. On the CIFAR10 dataset, it reduced the parameter count by 80.83% (from 2.3M to 0.55M) compared to the state-of-the-art WaveMix model, while simultaneously improving accuracy. On the more complex CIFAR100 dataset, LDAN outperformed previous non-learned hybrids by 6.32%.

Robustness to Rotation

Because Shearlets are designed to handle various rotations mathematically, LDAN is inherently more robust to image orientation changes. In tests involving 90°, 180°, and 270° rotations, LDAN maintained significantly higher accuracy than standard Wide Residual Networks (WRN). For instance, on CIFAR10 with a 90° rotation, LDAN's accuracy only dropped to 68.62%, whereas the WRN plummeted to 55.23%.

Feature Sparsity

The mathematical superiority of Shearlets is reflected in the sparsity of the extracted features. LDAN's feature maps had a sparsity ratio of 0.5671, compared to nearly 1.0 for Wavelet-based methods. This cleaner signal allows the subsequent neural layers to be shallower and more efficient.

Conclusion and Future Work

LDAN demonstrates that non-learned operators, when chosen correctly, can significantly enhance the efficiency and robustness of deep learning models. By leveraging the geometric advantages of Shearlets and a novel directional attention mechanism, the authors have created a model that is both lightweight and high-performing.

Future research will focus on automating the selection of hyperparameters, such as the number of Shearlet levels, perhaps through meta-learning. Additionally, the authors aim to test LDAN's performance on "in-the-wild" datasets characterized by complex lighting and occlusions.

Summary updated

Abstract

Due to the stable feature representation capability provided by non-learned operators, the integration with deep learning models, i.e., non-learned operator based deep learning models, has become a paradigm, however, performance-wise, it is still not promising. In this paper, by revisiting non-learned operator based deep learning models, we reveal the reasons for their underperformance: lack of geometric invariance, insufficient sparsity, and neglect of directional importance. In response, we present a Lightweight Directional-Aware Network (LDAN) for image classification. Specifically, to generate sparse geometric-invariant features, we propose a ShearletNet to capture multi-directional features in three different levels. Then, a Directional-Aware module is designed to highlight the discriminative multi-directional features and generate multi-scale features. Finally, a Pointwise Convolution module is used to integrate the multi-directional features with the multi-scale ones for reducing the computational resources. Experiments on the commonly used CIFAR10, CIFAR100, Self-Taught Learning 10 (STL10), and Tiny ImageNet datasets demonstrate the efficiency and effectiveness of the proposed LDAN. Compared to the existing non-learned operator based models, LDAN reduces the parameter count by 80.83% while achieving a 6.32% increase in accuracy.
BibTeX
@article{guo2025revisiting,
  author = {Yuwei Guo and Wenhao Zhang and Yupeng Gao and Licheng Jiao and Shuo Wang and Jiabo Du and Fang Liu},
  title = {{Revisiting non-learned operators based deep learning for image classification: a lightweight directional-aware network}},
  journal = {Artif. Intell. Rev.},
  year = {2025},
  doi = {10.1007/s10462-024-11038-0},
}
Figure