Efficient Distributed Inference of Deep Neural Networks via Restructuring and Pruning

Afshin Abdi, Saeed Rashidi, Faramarz Fekri, Tushar Krishna

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success but are increasingly outgrowing the memory and processing capabilities of individual edge devices. Distributed inference, which partitions a model across multiple workers, offers a solution. However, standard partitioning strategies often suffer from significant communication latency. In typical layers, every neuron may require inputs from all neurons in the preceding layer. When these neurons reside on different workers, the resulting inter-node communication becomes a bottleneck, often exceeding the time spent on actual computation.

The RePurpose Framework

Researchers from Georgia Tech have introduced RePurpose, a framework designed to minimize this communication overhead. Unlike naive pruning methods that simply delete connections and often degrade accuracy, RePurpose treats distributed inference as an optimization problem involving two primary steps: restructuring and pruning.

Restructuring via Neuron Permutation

The core innovation of RePurpose is the realization that the order of neurons within a hidden layer is arbitrary. By "shuffling" or permuting these neurons, the framework can group those with the strongest dependencies onto the same worker. This process effectively concentrates the computational workload within individual nodes, transforming a dense web of cross-worker communication into a sparse one.

Comparison between the original model (a) and the restructured model (b) partitioned across two workers.

To solve the combinatorial challenge of assigning thousands of neurons to multiple workers, RePurpose utilizes the Munkres (Hungarian) assignment algorithm. This allows for an efficient determination of the optimal "seating chart" for neurons to minimize total cross-worker communication costs.

Pruning and Fine-tuning

Following restructuring, the algorithm applies $\ell_0$ optimization to prune the remaining cross-worker edges that have the least impact on the model's output. While significant pruning initially reduces accuracy, the researchers found that a minimal amount of fine-tuning (1-10 epochs) is sufficient to restore performance to near-original levels.

Key Insights: Mathematical Formulation

The communication bottleneck in distributed inference is represented by the cross-terms in the layer equations. For a two-worker setup, the output $y$ is computed as: $y_1 = W_{11}^T x_1 + b_1 + W_{12}^T x_2$ $y_2 = W_{22}^T x_2 + b_2 + W_{21}^T x_1$

Mathematical representation of distributed inference highlighting the cross-worker communication terms ($W_{12}$ and $W_{21}$).

RePurpose aims to minimize the influence of the red terms ($W_{12}$ and $W_{21}$), which represent data that must be transferred between workers. By restructuring the weight matrices through permutation, the framework maximizes the magnitude of the diagonal blocks ($W_{11}$ and $W_{22}$), allowing the cross-worker terms to be pruned with minimal loss of information.

Experimental Evaluation

The framework was evaluated across three setups:

  • Synthetic: A fully connected network for 2D location classification.
  • MNIST: A LeNet-5 style CNN for distributed digit recognition.
  • CIFAR-10: A ResNet-like architecture used for object detection (specifically detecting "dogs") across multiple cameras.

Hardware performance was modeled using ASTRA-sim, simulating both high-bandwidth Datacenter and low-bandwidth Edge environments.

Results

RePurpose demonstrates significant improvements over baseline methods:

  • Extreme Sparsity: In MNIST tasks, the framework achieved original accuracy while utilizing less than 0.03% of the original cross-worker edges (only 10 to 30 connections out of 100,000).
  • Latency Reduction: In Edge network simulations, RePurpose provided up to a 5.7x speedup for MNIST and 2.8x for CIFAR-10. In Datacenters, speedups ranged from 1.7x to 10.47x depending on the sparsity level.
  • Efficiency: The method outperformed naive sparsification and model distillation, offering a superior trade-off between communication complexity and accuracy.

Limitations and Future Work

While effective for CNNs and fully connected layers, extending RePurpose to Transformers and their attention mechanisms remains a challenge. Additionally, the current approach assumes a static network topology; future iterations could explore dynamic restructuring to adapt to fluctuating worker availability or network conditions. Finally, the requirement for fine-tuning necessitates access to training data, which may be a constraint for certain pre-trained models.

Summary updated

BibTeX
@inproceedings{abdi2023efficient,
  author = {Afshin Abdi and Saeed Rashidi and Faramarz Fekri and Tushar Krishna},
  title = {{Efficient Distributed Inference of Deep Neural Networks via Restructuring and Pruning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i6.25815},
}
Figure