Abstract
Deep Neural Networks (DNNs) face significant deployment challenges on resource-constrained edge devices due to their substantial memory and computational requirements. While pruning is a common strategy to mitigate these issues, standard architectures are not inherently designed for extreme sparsity. This paper introduces Sparse Hybrid Linear-Morphological Networks, a novel architecture that integrates "max-plus" morphological operations into standard linear frameworks. By replacing traditional ReLU activations with explicitly sparse morphological layers, the researchers demonstrate that these hybrid models achieve state-of-the-art performance on audio and image classification tasks while maintaining significantly higher accuracy under aggressive pruning regimes compared to standard ReLU-based networks.
Introduction
The deployment of Deep Neural Networks (DNNs) on edge devices is often hindered by their size and complexity. Pruning—the process of removing redundant weights—is a standard technique used to compress these models. However, standard architectures often suffer from significant performance degradation when pruned beyond certain thresholds.
In contrast, Morphological Neural Networks (MNNs), which utilize max-plus and min-plus operations instead of the standard multiply-accumulate operations, are inherently sparse because they focus on extremal values. Historically, MNNs have been difficult to train due to the non-differentiable nature of their operations, often leading to poor convergence. This research, conducted by the National Technical University of Athens, proposes a hybrid approach that bridges the gap between the trainability of linear algebra and the structural benefits of mathematical morphology.
Methodology
The authors propose a hybrid structure that focuses on the classification "head" of the network, typically the fully connected layers. The architecture relies on three primary innovations:
1. Max-Plus Blocks
The researchers replace standard ReLU activations and maxout pooling with max-plus layers. Mathematically, a max-plus operation is defined as: $$x \boxplus w = \max(x_i + w_i)$$ This shift moves the network away from traditional linear combinations toward morphological operations that naturally select for dominant features.
2. Sparse Initialization
To overcome the training difficulties associated with dense morphological layers, the authors introduce a "Sparse Initialization" strategy. Most weights in the morphological layers are set to $-\infty$, rendering them inactive. Only a few active connections per neuron (typically $P=2$) are maintained. This explicit sparsity ensures the network remains efficient from the start of training.
3. Hybrid Topology
By inserting these sparse morphological layers between standard linear layers, the network retains the gradient-friendly properties of linear algebra while gaining the non-linear expressive power and structural sparsity of morphology.
Theoretical Insights
One of the paper's key theoretical contributions is the demonstration that the standard ReLU-activated layer is actually a specific, constrained case of a max-plus morphological layer. By relaxing these constraints, the hybrid network gains a broader expressive range.
Furthermore, the authors identify a "Sparsity Catalyst" effect. Because the morphological layer only processes maximum values, it induces sparsity in the preceding linear layers. The linear layers learn to produce a small number of high-quality signals rather than a large volume of redundant ones, as the subsequent morphological layer would ignore the lower-value signals anyway.
Experimental Results
The researchers evaluated the Sparse Hybrid Linear-Morphological Networks on two benchmarks: the Magna-Tag-A-Tune (MTAT) dataset for audio auto-tagging and the CIFAR-10 dataset for image classification.
Performance and Convergence
On the MTAT dataset, the Sparse-Morph model achieved a state-of-the-art ROC-AUC of 0.9152, slightly exceeding the performance of standard ReLU (0.9149) and Maxout (0.9148) baselines. Notably, despite the non-differentiable nature of the max operations, the hybrid models exhibited faster initial convergence than their standard counterparts.
Resilience to Pruning
The most significant advantage of the hybrid approach was observed during pruning experiments on CIFAR-10. When 90% of the parameters were pruned ($r_1=0.9, r_2=0.7$), the results were striking:
- Sparse-Morph Model: Maintained 70.31% accuracy.
- Standard ReLU Model: Accuracy dropped to 44.77%.
This demonstrates that the morphological layers provide a structural robustness that allows the network to retain its functional integrity even when the vast majority of its weights are removed.
Conclusion
Sparse Hybrid Linear-Morphological Networks offer a promising path forward for efficient deep learning. By blending the strengths of linear and morphological operations, the authors have created a framework that is both high-performing and exceptionally prunable. While this study focused on fully connected layers, future research may extend these sparse morphological principles to convolutional backbones and larger-scale datasets like ImageNet to further validate the scalability of the approach.