Introduction
Deep Neural Networks (DNNs) have achieved remarkable success but are increasingly outgrowing the memory and processing capabilities of individual edge devices. Distributed inference, which partitions a model across multiple workers, offers a solution. However, standard partitioning strategies often suffer from significant communication latency. In typical layers, every neuron may require inputs from all neurons in the preceding layer. When these neurons reside on different workers, the resulting inter-node communication becomes a bottleneck, often exceeding the time spent on actual computation.
The RePurpose Framework
Researchers from Georgia Tech have introduced RePurpose, a framework designed to minimize this communication overhead. Unlike naive pruning methods that simply delete connections and often degrade accuracy, RePurpose treats distributed inference as an optimization problem involving two primary steps: restructuring and pruning.
Restructuring via Neuron Permutation
The core innovation of RePurpose is the realization that the order of neurons within a hidden layer is arbitrary. By "shuffling" or permuting these neurons, the framework can group those with the strongest dependencies onto the same worker. This process effectively concentrates the computational workload within individual nodes, transforming a dense web of cross-worker communication into a sparse one.
To solve the combinatorial challenge of assigning thousands of neurons to multiple workers, RePurpose utilizes the Munkres (Hungarian) assignment algorithm. This allows for an efficient determination of the optimal "seating chart" for neurons to minimize total cross-worker communication costs.
Pruning and Fine-tuning
Following restructuring, the algorithm applies $\ell_0$ optimization to prune the remaining cross-worker edges that have the least impact on the model's output. While significant pruning initially reduces accuracy, the researchers found that a minimal amount of fine-tuning (1-10 epochs) is sufficient to restore performance to near-original levels.
Key Insights: Mathematical Formulation
The communication bottleneck in distributed inference is represented by the cross-terms in the layer equations. For a two-worker setup, the output $y$ is computed as: $y_1 = W_{11}^T x_1 + b_1 + W_{12}^T x_2$ $y_2 = W_{22}^T x_2 + b_2 + W_{21}^T x_1$
RePurpose aims to minimize the influence of the red terms ($W_{12}$ and $W_{21}$), which represent data that must be transferred between workers. By restructuring the weight matrices through permutation, the framework maximizes the magnitude of the diagonal blocks ($W_{11}$ and $W_{22}$), allowing the cross-worker terms to be pruned with minimal loss of information.
Experimental Evaluation
The framework was evaluated across three setups:
- Synthetic: A fully connected network for 2D location classification.
- MNIST: A LeNet-5 style CNN for distributed digit recognition.
- CIFAR-10: A ResNet-like architecture used for object detection (specifically detecting "dogs") across multiple cameras.
Hardware performance was modeled using ASTRA-sim, simulating both high-bandwidth Datacenter and low-bandwidth Edge environments.
Results
RePurpose demonstrates significant improvements over baseline methods:
- Extreme Sparsity: In MNIST tasks, the framework achieved original accuracy while utilizing less than 0.03% of the original cross-worker edges (only 10 to 30 connections out of 100,000).
- Latency Reduction: In Edge network simulations, RePurpose provided up to a 5.7x speedup for MNIST and 2.8x for CIFAR-10. In Datacenters, speedups ranged from 1.7x to 10.47x depending on the sparsity level.
- Efficiency: The method outperformed naive sparsification and model distillation, offering a superior trade-off between communication complexity and accuracy.
Limitations and Future Work
While effective for CNNs and fully connected layers, extending RePurpose to Transformers and their attention mechanisms remains a challenge. Additionally, the current approach assumes a static network topology; future iterations could explore dynamic restructuring to adapt to fluctuating worker availability or network conditions. Finally, the requirement for fine-tuning necessitates access to training data, which may be a constraint for certain pre-trained models.
Summary updated