Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs

Severin Bochem, Victor J. B. Jung, Arpan Suravi Prasad, Francesco Conti, Luca Benini

TL;DR

This paper introduces a distributed inference strategy that allows Small Language Models (SLMs) to run entirely on-chip across a network of low-power microcontrollers (MCUs). By partitioning Transformer layers without duplicating weights, the researchers achieved a 26.1× super-linear speedup on TinyLlama, effectively bypassing the "memory wall" that typically cripples edge AI performance.

What Problem Does This Paper Address?

Deploying Transformers on "extreme edge" devices—like smart glasses—is a nightmare for engineers. These devices use Micro-Controller Units (MCUs) that prioritize battery life over raw power.

The core challenge is the Memory Bottleneck. Even "Small" Language Models (SLMs) have tens of millions of parameters. Most MCUs have less than 8 MiB of on-chip memory (SRAM), meaning the model weights must be stored in off-chip memory (Flash or PSRAM). Accessing off-chip memory is orders of magnitude slower and more energy-intensive than on-chip access. Conventional distributed methods often replicate weights across chips or use "pipeline parallelism," which introduces high latency—unacceptable for real-time, interactive wearable AI.

Methods

The authors propose a tensor parallelism-based partitioning scheme specifically tailored for a multi-chip system of "Siracusa" SoCs (low-power RISC-V MCUs).

  1. Zero Weight Duplication: Unlike other edge-distribution methods, this approach ensures no model weights are replicated. Every byte of precious on-chip memory is used to store unique parameters.
  2. MHSA Partitioning: The Multi-Head Self-Attention (MHSA) layer is split across the "head" dimension. Each MCU handles a subset of attention heads independently.
  3. Fully-Connected (FC) Partitioning: The feed-forward layers are sliced across the intermediate dimension.
  4. Minimal Synchronization: The scheme is designed to require only two synchronization points (all-reduce operations) per Transformer block.
  5. Hierarchical Interconnect: To keep the system scalable, chips are organized in groups of four, using a MIPI serial interface for chip-to-chip communication to minimize energy costs (100 pJ/bit).

Datasets / Models

The researchers evaluated their methodology on two real-world Transformer architectures:

  • TinyLlama-42M: A decoder-only model (42 million parameters) tested in both autoregressive (token-by-token) and prompt (parallel) modes.
  • MobileBERT: An encoder-only model used for tasks like Natural Language Understanding.

Evaluation was performed using GVSoC, an event-driven simulator for RISC-V systems, combined with analytical power models.

Key Insights and Ideas

  • The "Super-Linear" Magic: The most striking finding is the super-linear speedup (e.g., getting a 26x boost from only 8 chips). This happens because, as the model is partitioned, the individual slices eventually become small enough to fit entirely within the fast L2 on-chip memory. This eliminates the massive "latency tax" of off-chip L3 memory access.
  • Memory-Bound vs. Compute-Bound: The paper highlights that Autoregressive mode (generating text) is heavily memory-bound, making it the biggest beneficiary of this distributed on-chip approach. In contrast, Prompt mode is compute-bound, where the benefits of distribution come from parallel processing rather than just memory locality.
  • Communication Efficiency: By carefully choosing where to split the tensors, the authors kept the inter-chip communication overhead low enough that it doesn't negate the gains from parallel execution.

Key Results

  • TinyLlama (Autoregressive): On a system of 8 MCUs, the model achieved a 26.1× speedup compared to a single-chip system. It reached a latency of 0.54 ms per inference with an energy consumption of just 0.64 mJ.
  • TinyLlama (Prompt): Achieved a 9.9× speedup on 8 chips, showing better-than-linear scaling.
  • MobileBERT: Using 4 MCUs resulted in a 4.7× speedup per chip without increasing the total energy per inference.
  • Scalability: In a scaled-up simulation with 64 MCUs, the system achieved a 60.1× speedup and a 1.3× reduction in energy for TinyLlama.

Limitations and Future Work

  • Simulation-Based: While GVSoC is a highly accurate architectural simulator, the results have not yet been demonstrated on a physical multi-chip PCB.
  • Kernel Scaling: The authors noted that as they increased the number of chips, the "GEMM" (General Matrix Multiply) kernels became so small that RISC-V core utilization dropped. This suggests a "diminishing returns" point where further partitioning might hurt efficiency.
  • Future Directions: The authors suggest exploring even more aggressive quantization and specialized hardware accelerators (like the N-EUREKA mentioned in the paper but not used in this study) to further drive down energy and latency.

Summary updated

BibTeX
@inproceedings{bochem2025distributed,
  author = {Severin Bochem and Victor J. B. Jung and Arpan Suravi Prasad and Francesco Conti and Luca Benini},
  title = {{Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs}},
  booktitle = {DATE},
  year = {2025},
  doi = {10.23919/date64628.2025.10992712},
}
Figure