EASTER: Learning to Split Transformers at the Edge Robustly

Xiaotian Guo, Quan Jiang, Yixian Shen, Andy D. Pimentel, Todor P. Stefanov

Large transformer models like GPT-4 and ViT have revolutionized AI, but their massive memory requirements—often reaching tens of gigabytes—make them nearly impossible to deploy on individual edge devices like IoT sensors or mobile boards. While distributing these models across a cluster of devices is a common workaround, edge environments are notoriously unstable. A single device failure due to battery loss or connectivity issues typically causes the entire inference process to collapse.

EASTER (Efficient and rAbust Split TransformER) is a novel framework designed to address this fragility. It enables the deployment of large transformers across multiple resource-constrained devices while ensuring the system remains functional even under severe hardware failures.

The Challenge: Fragile Distribution

In a traditional distributed setup, a model is partitioned into sequential blocks. If one device in the chain fails, the "missing" layer results in a total loss of information, causing accuracy to plummet to near-zero. Existing solutions often rely on expensive hardware redundancy or complex orchestration that exceeds the capabilities of low-power edge hardware.

The impact of device failure on a distributed transformer model and the resulting trade-off between accuracy and memory reduction.

The EASTER Methodology

EASTER introduces three core pillars to achieve robust, memory-efficient distribution:

1. Partial Split Strategy

Instead of splitting model blocks evenly, EASTER utilizes Taylor importance scores to rank every neuron by its contribution to the final output. It identifies a small fraction of "vital" neurons and replicates them across every device in the cluster. The remaining "less vital" neurons are then distributed. This ensures that even if several devices go offline, the most critical components of the model are preserved on the surviving hardware.

A comparison between traditional even splitting and EASTER’s partial split strategy, which replicates vital neurons (r) across all devices.

2. UCT-based Design Space Exploration (DSE)

Determining the optimal replication ratio for every layer in a massive model is a complex search problem. EASTER employs a variant of the Upper Confidence Bounds applied to Trees (UCT) algorithm, enhanced with SVM classifiers. This allows the system to intelligently learn which layers are most sensitive to failure and prioritize replication there, balancing the trade-off between memory savings and robustness.

3. Multinode Intermediate Representation (IR) Tool

To bridge the gap between theory and deployment, the authors developed a tool that automatically converts standard PyTorch models into a custom multinode format. This tool handles the complex communication primitives (such as AllGather and AllReduce) required to synchronize partial splits in real-time.

Experimental Results

The researchers evaluated EASTER using diverse architectures, including ViT-16 for image classification and GPT2-Large and Vicuna-7B for language tasks. Testing was conducted on clusters of NVIDIA Jetson Xavier NX boards.

  • Memory Efficiency: For the Vicuna-7B model, EASTER reduced the per-device memory requirement from 27 GB to approximately 9.24 GB—a 65.8% reduction—while maintaining functional reasoning capabilities.
  • Resilience to Failure: In a "3D-Fail" scenario (where 3 out of 4 devices fail), traditional partitioning methods saw accuracy drop to near-zero. In contrast, EASTER maintained usable performance. For ViT-16, EASTER reduced the Top-1 error from 94.7% (traditional) to 54.6% under the same failure conditions.
  • Latency Speedup: Distributing GPT2-Large across 8 devices reduced end-to-end inference latency from 58.0 seconds to just 7.62 seconds, demonstrating that distribution can provide significant speedups for large LLMs.

Conclusion and Future Work

EASTER demonstrates that redundancy does not have to be a waste of resources; rather, strategic redundancy can be a tunable parameter for edge robustness. By replicating only the top-tier "vital" connections, the framework prevents total system collapse during hardware failures.

Future research directions include adapting the "split" in real-time as devices dynamically join or leave the network and exploring "robustness-aware fine-tuning" to further recover accuracy lost during the partial splitting process.

Summary updated

Abstract

Prevalent large transformer models present significant computational challenges for resource-constrained devices at the Edge. While distributing the workload of deep learning models across multiple edge devices has been extensively studied, these works typically overlook the impact of failures of edge devices. Unpredictable failures, due to, e.g., connectivity issues or discharged batteries, can compromise the reliability of inference serving at the Edge. In this article, we introduce a novel methodology, called EASTER, designed to learn robust distribution strategies for transformer models against device failures that consider the tradeoff between robustness (i.e., maintaining model functionality against failures) and resource utilization (considering memory usage and computations). We evaluate EASTER with three representative transformers—ViT, GPT-2, and Vicuna—under device failures. Our results demonstrate EASTER’s efficiency in memory usage, and possible end-to-end latency improvement for inference across multiple edge devices while preserving model accuracy as much as possible under device failures.
BibTeX
@article{guo2024easter,
  author = {Xiaotian Guo and Quan Jiang and Yixian Shen and Andy D. Pimentel and Todor P. Stefanov},
  title = {{EASTER: Learning to Split Transformers at the Edge Robustly}},
  journal = {IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.},
  year = {2024},
  doi = {10.1109/tcad.2024.3438995},
}
Figure