Large transformer models like GPT-4 and ViT have revolutionized AI, but their massive memory requirements—often reaching tens of gigabytes—make them nearly impossible to deploy on individual edge devices like IoT sensors or mobile boards. While distributing these models across a cluster of devices is a common workaround, edge environments are notoriously unstable. A single device failure due to battery loss or connectivity issues typically causes the entire inference process to collapse.
EASTER (Efficient and rAbust Split TransformER) is a novel framework designed to address this fragility. It enables the deployment of large transformers across multiple resource-constrained devices while ensuring the system remains functional even under severe hardware failures.
The Challenge: Fragile Distribution
In a traditional distributed setup, a model is partitioned into sequential blocks. If one device in the chain fails, the "missing" layer results in a total loss of information, causing accuracy to plummet to near-zero. Existing solutions often rely on expensive hardware redundancy or complex orchestration that exceeds the capabilities of low-power edge hardware.
The EASTER Methodology
EASTER introduces three core pillars to achieve robust, memory-efficient distribution:
1. Partial Split Strategy
Instead of splitting model blocks evenly, EASTER utilizes Taylor importance scores to rank every neuron by its contribution to the final output. It identifies a small fraction of "vital" neurons and replicates them across every device in the cluster. The remaining "less vital" neurons are then distributed. This ensures that even if several devices go offline, the most critical components of the model are preserved on the surviving hardware.
2. UCT-based Design Space Exploration (DSE)
Determining the optimal replication ratio for every layer in a massive model is a complex search problem. EASTER employs a variant of the Upper Confidence Bounds applied to Trees (UCT) algorithm, enhanced with SVM classifiers. This allows the system to intelligently learn which layers are most sensitive to failure and prioritize replication there, balancing the trade-off between memory savings and robustness.
3. Multinode Intermediate Representation (IR) Tool
To bridge the gap between theory and deployment, the authors developed a tool that automatically converts standard PyTorch models into a custom multinode format. This tool handles the complex communication primitives (such as AllGather and AllReduce) required to synchronize partial splits in real-time.
Experimental Results
The researchers evaluated EASTER using diverse architectures, including ViT-16 for image classification and GPT2-Large and Vicuna-7B for language tasks. Testing was conducted on clusters of NVIDIA Jetson Xavier NX boards.
- Memory Efficiency: For the Vicuna-7B model, EASTER reduced the per-device memory requirement from 27 GB to approximately 9.24 GB—a 65.8% reduction—while maintaining functional reasoning capabilities.
- Resilience to Failure: In a "3D-Fail" scenario (where 3 out of 4 devices fail), traditional partitioning methods saw accuracy drop to near-zero. In contrast, EASTER maintained usable performance. For ViT-16, EASTER reduced the Top-1 error from 94.7% (traditional) to 54.6% under the same failure conditions.
- Latency Speedup: Distributing GPT2-Large across 8 devices reduced end-to-end inference latency from 58.0 seconds to just 7.62 seconds, demonstrating that distribution can provide significant speedups for large LLMs.
Conclusion and Future Work
EASTER demonstrates that redundancy does not have to be a waste of resources; rather, strategic redundancy can be a tunable parameter for edge robustness. By replicating only the top-tier "vital" connections, the framework prevents total system collapse during hardware failures.
Future research directions include adapting the "split" in real-time as devices dynamically join or leave the network and exploring "robustness-aware fine-tuning" to further recover accuracy lost during the partial splitting process.
Summary updated