USDN: A Unified Sample-Wise Dynamic Network With Mixed-Precision and Early-Exit

Ji-Ye Jeon, Xuan Truong Nguyen, Soojung Ryu, Hyuk-Jae LeeWACV 2024

TL;DR

This paper introduces USDN, a novel neural network architecture that dynamically adapts its computation based on input complexity, using a combination of mixed-precision quantization and early-exit mechanisms. USDN achieves state-of-the-art performance on ImageNet by intelligently allocating computational resources, reducing inference costs by up to 12.78% without sacrificing accuracy.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are computationally expensive, making them challenging to deploy on resource-constrained edge devices. While quantization techniques reduce model size and computation, they often apply a uniform bit-width across all layers, leading to inefficiencies. Dynamic networks, which adapt their execution path based on input complexity, offer a promising solution. However, combining dynamic execution with mixed-precision quantization (where different layers have different bit-widths) presents a significant challenge. This is because jointly optimizing the locations of early-exit classifiers (ICs) and layer-wise bit-configurations is a complex search problem that can lead to accuracy degradation and increased training time. This paper tackles the problem of creating a unified, fully quantized, input-adaptive network that efficiently balances accuracy and computational cost.

Methods

The core of USDN is a unified supernet that encompasses both mixed-precision configurations and early-exit points. Instead of relying on conventional weight-sharing supernets, USDN uses dedicated model parameters for different bit-widths during training. This approach avoids the accuracy degradation often seen with shared weights.

Key methodological contributions include:

  • Scale-aware IC Design: To address accuracy issues with early exits on shallow feature maps, USDN incorporates scale-aware internal classifiers. These ICs adjust their internal structure (e.g., adding stride-2 convolutions) to match the input feature map size, ensuring more reliable predictions at earlier stages.
  • Compound Architectural Search: USDN introduces a novel objective function for its reinforcement learning (RL)-based search. This function balances the "inference cost for successful exit rates" with a "failure cost for failed exit attempts," allowing the network to learn optimal IC locations and layer-wise bit-widths simultaneously.
  • IC-wise Knowledge Distillation (KD): After the architectural search, a full-precision teacher network is used to guide the training of the quantized student network. This KD process is applied at the IC level, transferring knowledge from the high-precision teacher's classifiers to the quantized ones, further improving accuracy.

Datasets

  • ImageNet: Used for training and evaluation, with 1.2 million training images and 50,000 validation/test images. For the search process, 50,000 images from the training set were used as a validation set.
  • CIFAR-10: Used for experiments on a smaller network (ResNet20) to compare with fixed-precision methods.
  • CIFAR-100: Used for evaluating USDN on MobileNetV1 baseline against uniformly quantized SDN.

Key Insights and Ideas

  • Joint Optimization is Crucial: The paper highlights that simply combining existing mixed-precision and early-exit techniques is not sufficient. USDN's success stems from its unified approach to searching for both IC locations and bit-widths simultaneously.
  • Scale-Awareness for Early Exits: The proposed scale-aware IC design is a critical insight. It directly addresses the problem of unreliable early predictions when feature maps are large, a common issue in early-exit networks.
  • Dedicated Parameters for Mixed Precision: Using dedicated parameters for different bit-widths during training, rather than weight sharing, proves more effective for maintaining accuracy in mixed-precision networks.
  • KD at the IC Level: Applying knowledge distillation specifically to the internal classifiers (ICs) is more effective than general self-distillation for quantized early-exit networks.

Key Results

  • ImageNet Performance: USDN achieves state-of-the-art performance on ImageNet, reducing computational cost by 12.78% compared to previous mixed-precision methods without accuracy degradation. It also shows a 0.68% absolute gain in accuracy compared to input-adaptive quantization methods at a similar computational cost.
  • CIFAR-10 Performance: USDN demonstrates significant cost reduction in terms of BOPs while preserving high accuracy compared to fixed-precision methods.
  • CIFAR-100 Performance: USDN achieves +1.1% higher accuracy than uniformly quantized 4-bit SDN at the same computation cost, despite having fewer ICs.
  • Computational Efficiency: The ICs in USDN account for only 2.69% of the overall computation cost, showcasing their efficiency.

Limitations and Future Work

  • Training Time: While USDN aims to reduce inference cost, the architectural search process itself can be computationally intensive due to the exploration of a large design space.
  • Generalizability of KD: The paper notes that self-distillation was ineffective for quantized EEnets, suggesting that the effectiveness of KD might be sensitive to the specific architecture and quantization scheme.
  • Future Work: The authors suggest further exploration of the trade-off ratio for different applications and datasets. Investigating more advanced search strategies and exploring the application of USDN to other network architectures and tasks are also potential future directions.

Summary updated

BibTeX
@inproceedings{jeon2024usdn,
  author = {Ji-Ye Jeon and Xuan Truong Nguyen and Soojung Ryu and Hyuk-Jae Lee},
  title = {{USDN: A Unified Sample-Wise Dynamic Network With Mixed-Precision and Early-Exit}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure