Abstract
Deep neural networks achieve state-of-the-art performance but are often too computationally expensive for edge deployment. This paper introduces a principled framework to compress deep networks into shallow, efficient versions by treating the layer-wise transformation as a "straight flow." By distilling a deep teacher into a continuous-time neural flow and subsequently "straightening" its trajectory, the authors compress complex multi-layer operations into a single high-performing block. This approach maintains high accuracy while significantly reducing computational costs across various architectures.
Introduction
While depth is a primary driver of success in modern neural networks, it introduces significant latency and memory overhead, hindering deployment on resource-constrained devices like IoT sensors and smartphones. Traditional compression techniques, such as Knowledge Distillation (KD) or pruning, often suffer from a "performance cliff" when the student network is substantially shallower than the teacher.
Weight-sharing networks offer a potential solution by reusing layers, but they are difficult to train and lack a principled mechanism for reducing inference iterations. This research addresses these challenges by framing network depth as a transport problem, allowing for the conversion of deep, multi-layer models into efficient one-step models without the typical loss in precision.
Methodology
The proposed framework utilizes a three-stage training pipeline to transform a deep teacher network into a shallow student:
- Continuous-Time Flow Distillation: The deep teacher is first distilled into a time-continuous neural flow model (a Neural ODE). This model learns to map inputs to the teacher's outputs through infinitesimal steps.
- ReFlow (Straightening): Continuous flows often follow complex, "curvy" paths that are difficult to approximate in few steps. The authors apply the ReFlow operator to straighten the transport trajectory. By enforcing a constant-speed, straight-line movement, the entire transformation becomes easier to simulate in just one or two steps.
- One-Step Distillation & Refinement: The straightened flow is compressed into a final one-step model. Standard KD is used to fine-tune this single-block student, ensuring it captures the teacher's performance in a single forward pass.
Key Insights
- Transport Cost and Straightness: The difficulty of compression is fundamentally linked to the "transport cost" of the network's internal transformations. Minimizing this cost via straightening allows a single layer to perform the work of many.
- Theory-Guided Weight-Sharing: This method provides a mathematical foundation for reducing simulation steps in flow models, which is equivalent to reducing the number of blocks in a physical network.
- Dynamic Inference: The flow-based nature of the model naturally supports early-exiting. Simple tasks can be completed in one step, while harder inputs can run for additional steps to improve accuracy, providing a flexible speed-accuracy trade-off.
Experimental Results
The method was evaluated on ImageNet-1K and CIFAR-10 using Vision Transformers (SWIN, DeiT) and ResNet-50.
As shown in the results table, the proposed method consistently outperforms standard KD and pruning baselines:
- SWIN-Base: The model achieved 82.6% top-1 accuracy on ImageNet with only 5.1G FLOPs—a 66% reduction in computational cost compared to the original 15.1G FLOPs, with only a marginal 0.5% drop in accuracy.
- ResNet-50: The framework compressed ResNet-50 to a version with 55% fewer FLOPs while maintaining the original 77.0% accuracy.
- Semantic Segmentation: On the ADE20K dataset, a two-step model matched the mIoU of a standard SWIN-Base backbone while increasing inference speed from 8.7 to 10.5 FPS.
Conclusion and Future Work
This framework demonstrates that straightening the internal trajectories of neural networks is a powerful tool for layer compression. While highly effective, the current approach requires input and output resolutions to match within compressed blocks, necessitating separate flow models for multi-resolution architectures like standard ResNets. Future research may explore Neural Architecture Search (NAS) to automatically optimize the structure of these flow blocks and further streamline the multi-stage training process.
Summary updated