Abstract
Convolutional neural networks (CNNs) rely on the depth of the architecture to obtain complex features. It results in computationally expensive models for low-resource IoT devices. Convolutional operators are local and restricted in the receptive field, which increases with depth. We explore partial differential equations (PDEs) that offer a global receptive field without the added overhead of maintaining large kernel convolutional filters. We propose a new feature layer, called the Global layer, that enforces PDE constraints on the feature maps, resulting in rich features. These constraints are solved by embedding iterative schemes in the network. The proposed layer can be embedded in any deep CNN to transform it into a shallower network. Thus, resulting in compact and computationally efficient architectures achieving similar performance as the original network. Our experimental evaluation demonstrates that architectures with global layers require 2-5x less computational and storage budget without any significant loss in performance.
Introduction
Modern CNNs achieve high performance by stacking many layers to gradually increase the "receptive field"—the area of the input image that influences a specific feature. However, this depth makes models computationally expensive and memory-intensive, which is a major hurdle for deployment on low-resource IoT devices. While previous works explored "Neural ODEs" (Ordinary Differential Equations) to model continuous feature evolution, these often require expensive fixed-point solvers that negate their theoretical efficiency. This paper seeks a way to capture global spatial relationships and complex feature compositions without the overhead of extreme depth or large-kernel filters.
Methods
The core innovation is the Global feature layer, which enforces a PDE constraint on feature maps.
- The PDE Framework: Instead of simple discrete steps, the layer treats feature evolution as an advection-diffusion process. In physics, this describes how particles move (advection) and spread out (diffusion). Here, it allows pixels to interact globally across the feature map.
- Condensing Depth: In a standard CNN, a specific block might be repeated $m$ times (e.g., 5 times in ResNet32). The authors replace $m-1$ of these blocks with one Global layer. This "condenses" the network's depth while retaining the "signature" of the original architecture.
- Efficient Solver: To make this practical, they embed a simple iterative finite-difference solver directly into the network. This solver runs for a fixed number of steps ($K$), making the computational cost predictable and much lower than traditional black-box ODE solvers.
- Hybrid Approach: The layer is "plug-and-play." It uses one original convolutional block to initialize the PDE solution, ensuring the model still benefits from the specific design of the base architecture (like ResNet or MobileNet).
Experimental Evaluation
The authors evaluated their approach on standard computer vision benchmarks including MNIST-10, CIFAR-10, CIFAR-100, and ImageNet-1000 (ILSVRC 2012). The focus was on improving efficiency on these established benchmarks without introducing new datasets.
Key Results
The "Global" versions of popular architectures showed massive efficiency gains:
- ResNet32 (CIFAR-10): The condensed version (ResNet32-Global) achieved 91.93% accuracy using only 15M MACs (multiply-accumulate operations), compared to the original's 92.49% accuracy at 70M MACs. This represents a ~4.7× reduction in compute.
- Storage Savings: On MNIST, the ResNet-Global model required 3–5× fewer parameters than the baseline.
- ImageNet Performance: For MobileNetV3-Global, they achieved similar Top-1 accuracy to the baseline while reducing parameters by 1.8× and MACs by 1.4×.
- Speed: Inference and training times were significantly reduced. For example, ResNet56-Global's inference was ~2.8× faster than the standard ResNet56 on a V100 GPU.
Discussion and Insights
Unlike standard convolutions which are local (e.g., $3 \times 3$ pixels), the diffusion term in a PDE allows information to propagate across the entire image, effectively providing an infinite receptive field in a shallow network. The paper argues that much of the depth in modern CNNs is redundant; a single layer solving a well-defined physical equation can replace many layers of "brute-force" convolutions. By using one original block to "start" the PDE solver, the network converges much faster and maintains the performance characteristics of the parent architecture.
Limitations and Future Work
The current implementation handles image boundaries by "rolling" the image, though more sophisticated boundary condition designs could improve results. Additionally, while many parameters are learned, some (like the time step $\delta_t$) are currently treated as hyperparameters. Future work could involve using Neural Architecture Search (NAS) to find optimal PDE configurations automatically or exploring other PDEs like the Wave equation or Navier-Stokes for different types of data tasks.
Summary updated