Abstract
Manual design of efficient Deep Neural Networks (DNNs) for mobile and edge devices is an involved process which requires expert human knowledge to improve efficiency in different dimensions. In this paper, we present DEff-ARTS, a differentiable efficient architecture search method for automatically deriving CNN architectures for resource constrained devices. We frame the search as a multi-objective optimization problem where we minimize the classification loss and the computational complexity of performing inference on the target hardware. Our formulation allows for easy trading-off between the sub-objectives depending on user requirements. Experimental results on CIFAR-10 classification showed that our approach achieved a highly competitive test error rate of 3.24% with 30% fewer parameters and multiply and accumulate (MAC) operations compared to Differentiable ARchiTecture Search (DARTS).
Introduction
Deploying Deep Neural Networks (DNNs) on mobile phones and embedded sensors is a balancing act. High accuracy is desired, but strict hardware constraints like battery life and processing power impose significant limitations. Traditionally, engineers have relied on hand-crafted efficient architectures or automated search (NAS) methods. However, early NAS methods were computationally expensive, and newer differentiable methods like DARTS often use poor proxies for efficiency, such as FLOPs (Floating Point Operations).
The problem with FLOPs is their inconsistent correlation with real-world speed. For example, on many processors, a division operation is significantly more resource-intensive than an addition, despite both counting as a single "operation." DEff-ARTS addresses this by incorporating actual hardware latency—specifically CPU cycles—directly into the automated search process.
Methodology
DEff-ARTS builds upon DARTS (Differentiable ARchiTecture Search), treating the architecture search as a continuous optimization problem solvable via gradient descent. The method introduces three key innovations:
- Differentiable Compute Cost ($L_{com}$): A mathematical formula estimates the total CPU cycles required for a network. The authors specifically modeled the cycle requirements for a Texas Instruments C64x+ processor, accounting for the higher cost of operations like "Average Pooling" compared to "Max Pooling."
- Multi-Objective Optimization: Standard classification loss is combined with this compute cost into a single "Performance Loss." This allows the algorithm to penalize architectures that are computationally inefficient.
- Non-Linear Transformation: To handle the different scales of accuracy and cost, the authors applied an exponentiation trick to the compute cost. This "linearizes" the relationship, preventing expensive operations from dominating the search too early and enabling a smooth trade-off.
One critical insight is the use of a modulation parameter ($\beta$). By exponentiating the cost, the gradient for a single operation becomes dependent on the entire cell structure, leading to more holistic architectural choices rather than just local optimizations.
Experimental Results
The researchers evaluated DEff-ARTS using the CIFAR-10 dataset. The experimental results demonstrate a clear "win-win" for efficiency:
- Competitive Accuracy: DEff-ARTS achieved a test error of 3.24%, very close to the 2.76% of the much larger standard DARTS model.
- Significant Compression: The resulting models had 30% fewer parameters and 30% fewer MACs than DARTS.
- Hardware Efficiency: Depending on the weighting of cost vs. accuracy, the models reduced actual compute cost (CPU cycles) by 1.9x to 4.5x.
The authors successfully demonstrated a Pareto front, showing that users can tune a single hyperparameter ($\Gamma$) to prioritize either maximum accuracy or maximum efficiency.
Conclusion and Future Work
DEff-ARTS provides a fast, hardware-aware search method that finds efficient models in roughly 4 GPU days—three orders of magnitude faster than older reinforcement learning approaches. While the current implementation is tied to specific Texas Instruments processor cycle counts, the method is generalizable to other hardware by recalculating instruction costs.
Future work could expand the search space to include more diverse operations and explicitly account for memory bandwidth or cache misses, which are critical bottlenecks in edge AI deployment.
Summary updated