State-of-the-art deep learning models are notoriously power-hungry and memory-intensive. While deploying these models in the cloud is straightforward, porting them to edge hardware—such as microcontrollers, IoT sensors, and mobile devices—presents a severe bottleneck. These decentralized devices often operate with less than 1MB of RAM and strict battery budgets, making standard architectures impossible to execute locally.
To bridge this gap, researchers have developed a variety of model compression and architectural optimization strategies. Rather than treating hardware constraints as an afterthought, these techniques redesign how neural networks store information and compute representations, allowing compact models to retain high performance.
Exploiting Parameter Redundancy via Weight Hashing
Deep neural networks are fundamentally overparameterized; they contain far more unique weights than necessary to represent the underlying function. One intuitive way to compress a network is to force multiple connections to share the exact same weight value.
In Compressing Neural Networks with the Hashing Trick, Wenlin Chen et al. demonstrated that this parameter sharing can be automated using a low-cost hash function. Instead of storing a massive weight matrix, the network stores a tiny pool of unique weight values. During the forward pass, a hash function maps the coordinates of the large virtual matrix to an index in the small physical pool. This allows the model to maintain a large virtual architecture—preserving its capacity to learn complex features—while drastically shrinking its actual memory footprint.
Pruning the Dead Weight
If weight sharing reduces the variety of parameters, pruning eliminates unnecessary parameters entirely. The intuition is simple: if a specific connection or neuron contributes very little to the final prediction, it can be zeroed out and ignored during computation.
Traditional pruning follows a clumsy three-step cycle: train a large model, remove low-magnitude weights, and fine-tune the remaining sparse network. To streamline this process, Manoj Rohit Vemparala et al. introduced an alternative in Adversarial Robust Model Compression Using In-Train Pruning. By learning "pruning masks" dynamically during the training process itself, the authors bypass the suboptimal post-processing cycle, yielding highly compact networks that maintain robust performance.
Taking a broader architectural view, Gihun Lee et al. developed the SIPA: A Simple Framework for Efficient Networks framework, which systematically combines neural architecture search with pruning. This multi-stage optimization ensures that the underlying network topology is inherently tailored to the memory and battery constraints of IoT sensors before deployment.
Extreme Quantization and Binarization
Standard neural networks represent weights and activations using 32-bit floating-point numbers (FP32). On a desktop GPU, multiplying these high-precision decimals is trivial. On an edge microcontroller, however, floating-point math is incredibly expensive in terms of both clock cycles and electrical energy.
Quantization solves this by mapping continuous FP32 values to discrete, lower-bit representations, such as 8-bit integers (INT8). As explored by Haotong Qin in Hardware-friendly Deep Learning by Network Quantization and Binarization, this concept can be pushed to its logical extreme: Binary Neural Networks (BNNs).
In a BNN, weights and activations are restricted to just 1 bit: either $+1$ or $-1$. This constraint completely redefines how the hardware computes. Instead of performing resource-intensive floating-point multiplications, the hardware can execute layers using native bitwise XNOR and popcount operations. This shifts the computational burden from complex arithmetic units to simple logic gates, unlocking massive storage savings and accelerating inference on tiny edge nodes.
Distillation and Split Processing
When a model cannot be compressed enough to fit entirely on a single edge device, researchers look to alternative paradigms that distribute the intelligence.
One approach is knowledge distillation, where a large, high-performance "teacher" model transfers its learned representations to a lightweight "student" model. The student network learns to mimic the teacher's output probabilities rather than learning from raw labels alone, capturing complex decision boundaries with a fraction of the parameters. For instance, Yuhan Long et al. leveraged this in Two-Stage Edge-Side Fault Diagnosis Method Based on Double Knowledge Distillation to compress diagnostic tools down to the strict limits of industrial IoT hardware.
When local execution remains completely impossible, the network can be physically partitioned. Guangyu Zhu et al. proposed a cooperative framework in ESFL: Efficient Split Federated Learning over Resource-Constrained Heterogeneous Wireless Devices that dynamically splits a neural network between an edge device and a central server. By shifting the "split point" based on the device's real-time battery level and network bandwidth, the system balances local computational strain with communication overhead, ensuring reliable execution across heterogeneous hardware environments.
Key Foundational Work
Compressing Neural Networks with the Hashing Trick (2015) — contributed 1 concept elements to this direction.
Preface (2014) — contributed 1 concept elements to this direction.