Layer Compression of Deep Networks with Straight Flows

Chengyue Gong, Xiaocong Du, Bhargav Bhushanam, Lemeng Wu, Xingchao Liu, Dhruv Choudhary, Arun Kejariwal, Qiang LiuAAAI 2024 · OpenReview

Abstract

Deep neural networks achieve state-of-the-art performance but are often too computationally expensive for edge deployment. This paper introduces a principled framework to compress deep networks into shallow, efficient versions by treating the layer-wise transformation as a "straight flow." By distilling a deep teacher into a continuous-time neural flow and subsequently "straightening" its trajectory, the authors compress complex multi-layer operations into a single high-performing block. This approach maintains high accuracy while significantly reducing computational costs across various architectures.

Introduction

While depth is a primary driver of success in modern neural networks, it introduces significant latency and memory overhead, hindering deployment on resource-constrained devices like IoT sensors and smartphones. Traditional compression techniques, such as Knowledge Distillation (KD) or pruning, often suffer from a "performance cliff" when the student network is substantially shallower than the teacher.

Weight-sharing networks offer a potential solution by reusing layers, but they are difficult to train and lack a principled mechanism for reducing inference iterations. This research addresses these challenges by framing network depth as a transport problem, allowing for the conversion of deep, multi-layer models into efficient one-step models without the typical loss in precision.

Methodology

The proposed framework utilizes a three-stage training pipeline to transform a deep teacher network into a shallow student:

  1. Continuous-Time Flow Distillation: The deep teacher is first distilled into a time-continuous neural flow model (a Neural ODE). This model learns to map inputs to the teacher's outputs through infinitesimal steps.
  2. ReFlow (Straightening): Continuous flows often follow complex, "curvy" paths that are difficult to approximate in few steps. The authors apply the ReFlow operator to straighten the transport trajectory. By enforcing a constant-speed, straight-line movement, the entire transformation becomes easier to simulate in just one or two steps.
  3. One-Step Distillation & Refinement: The straightened flow is compressed into a final one-step model. Standard KD is used to fine-tune this single-block student, ensuring it captures the teacher's performance in a single forward pass.
The three-stage framework: (a) Original network, (b) Flow distillation, (c) ReFlow straightening, and (d) Final one-step distillation.

Key Insights

  • Transport Cost and Straightness: The difficulty of compression is fundamentally linked to the "transport cost" of the network's internal transformations. Minimizing this cost via straightening allows a single layer to perform the work of many.
  • Theory-Guided Weight-Sharing: This method provides a mathematical foundation for reducing simulation steps in flow models, which is equivalent to reducing the number of blocks in a physical network.
  • Dynamic Inference: The flow-based nature of the model naturally supports early-exiting. Simple tasks can be completed in one step, while harder inputs can run for additional steps to improve accuracy, providing a flexible speed-accuracy trade-off.

Experimental Results

The method was evaluated on ImageNet-1K and CIFAR-10 using Vision Transformers (SWIN, DeiT) and ResNet-50.

Performance comparison across SWIN, DeiT, and ResNet-50 architectures, showing FLOPs reduction and Top-1 Accuracy.

As shown in the results table, the proposed method consistently outperforms standard KD and pruning baselines:

  • SWIN-Base: The model achieved 82.6% top-1 accuracy on ImageNet with only 5.1G FLOPs—a 66% reduction in computational cost compared to the original 15.1G FLOPs, with only a marginal 0.5% drop in accuracy.
  • ResNet-50: The framework compressed ResNet-50 to a version with 55% fewer FLOPs while maintaining the original 77.0% accuracy.
  • Semantic Segmentation: On the ADE20K dataset, a two-step model matched the mIoU of a standard SWIN-Base backbone while increasing inference speed from 8.7 to 10.5 FPS.

Conclusion and Future Work

This framework demonstrates that straightening the internal trajectories of neural networks is a powerful tool for layer compression. While highly effective, the current approach requires input and output resolutions to match within compressed blocks, necessitating separate flow models for multi-resolution architectures like standard ResNets. Future research may explore Neural Architecture Search (NAS) to automatically optimize the structure of these flow blocks and further streamline the multi-stage training process.

Summary updated

BibTeX
@inproceedings{gong2024layer,
  author = {Chengyue Gong and Xiaocong Du and Bhargav Bhushanam and Lemeng Wu and Xingchao Liu and Dhruv Choudhary and Arun Kejariwal and Qiang Liu},
  title = {{Layer Compression of Deep Networks with Straight Flows}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=4c3tSz6paL},
}
Figure