When deploying deep learning models to the real world, developers encounter a harsh reality: the hardware ecosystem is deeply fragmented. A model designed and optimized on a powerful server GPU must often run on a chaotic mix of smartphones, smart speakers, wearables, and embedded microcontrollers.
Traditionally, deep learning models have been evaluated using theoretical metrics like parameter counts or Floating-Point Operations (FLOPs). However, as Peiyan Dong et al. point out, theoretical computation does not map directly to real-world latency. A network that runs efficiently on one chip may crawl on another. To bridge this gap, the field of cross-platform model adaptation focuses on building neural networks that can fluidly adapt to the unique constraints of heterogeneous hardware.
The Root of Performance Fragmentation
Why does a single model architecture perform so inconsistently across different devices? The variance stems from fundamental architectural differences in the underlying silicon.
As Jonathan Huang et al. and Xubin Wang et al. observe, different hardware platforms—such as Central Processing Units (CPUs), Graphics Processing Units (GPUs), Digital Signal Processors (DSPs), and Neural Processing Units (NPUs)—rely on vastly different memory bandwidths, cache sizes, and native instruction sets. For example, an operation that is highly parallelized and hardware-accelerated on a desktop GPU might lack kernel support on a mobile NPU, forcing the runtime engine to fall back to a slow CPU implementation.
Furthermore, mobile chipsets exhibit significant fragmentation in how they handle numerical precision. As highlighted by the AI Benchmark study, some platforms prioritize floating-point performance, while others are optimized almost exclusively for quantized integer operations. Because of these hardware-dependent traits, Jin-Dong Dong et al. argue that it is impossible to find a single, universally optimal neural network architecture for all devices.
The naive solution to this problem is to treat every device profile as a completely separate deployment target, manually redesigning or fine-tuning a bespoke model for every chipset. However, when deploying to a fleet of thousands of distinct device types, this approach becomes computationally and logistically ruinous (Achintya Kundu et al.; Jiahui Yu). Every time the core model architecture or training data updates, engineers would have to retrain and redeploy every specialized version individually to meet specific latency requirements (Qiang Wang 0050 et al.).
To scale AI to the edge, researchers have developed techniques to decouple model training from the specific hardware constraints of the deployment target.
Train Once, Search Everywhere: Super-Networks
Instead of training thousands of independent models, modern hardware-aware deployment frameworks train a single, massive Super-Network that contains thousands of smaller, valid sub-networks within its weights. This is the core philosophy behind the "Once-For-All" (OFA) paradigm.
Super-Network (Contains all weights)
└── Sub-Network A (Max Depth/Width) --> Flagship Mobile GPU
└── Sub-Network B (Mid Depth/Width) --> Budget Smartphone NPU
└── Sub-Network C (Min Depth/Width) --> Low-Power IoT Sensor
The intuition behind why this works lies in weight inheritance and progressive shrinking. The largest network configuration is trained first to establish a strong baseline of feature representations. Smaller sub-networks are then gradually joint-trained by sampling paths within the larger network and inheriting its weights. This weight-sharing mechanism ensures that the sub-networks do not need to be trained from scratch, drastically lowering the computational barrier. Manas Sahni demonstrated that this massive search space can be further streamlined using compound scaling heuristics, which prune away sub-optimal configurations and accelerate multi-platform deployment.
However, a major challenge remains: how do we know which sub-network will run best on a new, unseen device without physically benchmarking every single option?
Bingqian Lu et al. solved this by exploiting a property known as latency monotonicity. They discovered that the relative execution speeds of different network architectures are highly correlated across diverse hardware platforms. If Architecture A is faster than Architecture B on a proxy device, it is highly likely to be faster on the target device as well. By leveraging a single proxy device and applying rapid transfer learning with just a handful of physical measurements, developers can accurately predict performance across an entire fleet of hardware without exhaustive profiling.
Runtime Adaptation: Slimmable and Flexible Networks
Super-networks solve the deployment problem at compile-time, but hardware environments are also highly dynamic at runtime. A smartphone might experience severe thermal throttling, or a sudden spike in background tasks might instantly deplete available memory (Stylianos Venieris et al.).
To handle these fluctuations without crashing or dropping frames, models must be able to adapt their computational footprint on the fly. This need led to the creation of Slimmable Neural Networks.
Jiahui Yu demonstrated that a single neural network can be executed at multiple different channel widths dynamically. The core mechanism that makes this possible is Switchable Batch Normalization (SBN). In a standard network, changing the layer width on the fly would alter the mean and variance of internal feature activations, causing the model's accuracy to collapse. SBN solves this by maintaining independent batch normalization statistics for each supported width configuration. At runtime, the model simply switches to the pre-computed normalization statistics corresponding to its current width, allowing instant execution scaling based on immediate hardware constraints.
This flexibility can also be applied to a network's depth. Jaesong Lee et al. and Qiang Wang 0050 et al. showed that models can be pruned or shortened on demand during inference by leveraging intermediate loss functions during training, such as Connectionist Temporal Classification (CTC) or multi-task objectives. By forcing intermediate layers to learn how to generate valid outputs, the model can instantly drop its tail layers to meet strict runtime latency budgets on lower-end hardware, sacrificing a small fraction of accuracy for a massive boost in speed.
Cross-Stack Execution and Backend Abstraction
Adapting the model architecture is only half the battle; the software execution layer must also adapt to the underlying silicon.
┌────────────────────────────────────────────────────────┐
│ Adaptive DNN Model │
│ (Slimmable Width, Variable Layer Depth) │
├────────────────────────────────────────────────────────┤
│ Inference Engine (e.g., MNN) │
│ (Pre-inference Optimization & Backend Abstraction) │
├────────────────────────────────────────────────────────┤
│ Inter-Operator Scheduler │
│ (Parallel Operator Execution & Batch Tuning) │
├────────────────────────────────────────────────────────┤
│ Heterogeneous Hardware │
│ [CPU] [GPU] [DSP] [NPU] │
└────────────────────────────────────────────────────────┘
Unified mobile inference engines, such as MNN developed by Xiaotang Jiang et al., introduce backend abstraction layers and pre-inference optimizations. These engines decouple the model's computational graph from the hardware-specific execution code, ensuring that the same underlying model can execute seamlessly across different hardware vendors with minimal manual engineering.
Finally, execution efficiency can be maximized by optimizing how operations are scheduled within the chip itself. Ligeng Zhu demonstrated that moving beyond standard intra-operator optimizations to embrace inter-operator parallelism—scheduling multiple independent operators to execute concurrently—can yield significant speedups. By dynamically adjusting scheduling strategies based on the specific batch sizes and hardware concurrency limits of the target platform, execution engines can extract maximum throughput from highly fragmented hardware environments.
Key Foundational Work
DPP-Net: Device-Aware Progressive Search for Pareto-Optimal Neural Architectures (2018) — contributed 1 concept elements to this direction.
Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors (2017) — contributed 1 concept elements to this direction.