Introduction
Deploying Large Language Models (LLMs) or even Small Language Models (SLMs) on "extreme edge" devices, such as smart glasses or wearable sensors, presents a significant engineering challenge. These devices typically rely on Micro-Controller Units (MCUs) that prioritize energy efficiency over raw performance. The primary obstacle is the Memory Wall: SLMs often require tens of millions of parameters, far exceeding the typical 8 MiB of on-chip SRAM available in modern MCUs. Consequently, model weights must be stored in off-chip Flash or PSRAM, where access is orders of magnitude slower and more energy-intensive than on-chip access.
This paper proposes a distributed inference strategy designed to run Transformers entirely on-chip across a network of low-power RISC-V MCUs, effectively bypassing the memory bottleneck by partitioning layers without duplicating weights.
Proposed Partitioning Strategy
The researchers developed a tensor parallelism-based partitioning scheme specifically for multi-chip systems of "Siracusa" SoCs. The strategy focuses on Zero Weight Duplication, ensuring that every byte of precious on-chip memory is used to store unique parameters rather than redundant copies.
As shown in the figure above, the partitioning is applied to two main components of the Transformer block:
- Multi-Head Self-Attention (MHSA): The layer is split across the "head" dimension, allowing each MCU to process a subset of attention heads independently.
- Fully-Connected (FC) Layers: The feed-forward network is sliced across the intermediate dimension.
This approach requires only two synchronization points (all-reduce operations) per Transformer block, minimizing communication overhead. To maintain scalability, the chips are organized into hierarchical groups connected via low-power MIPI serial interfaces (100 pJ/bit).
Experimental Setup
The methodology was evaluated on two representative architectures:
- TinyLlama-42M: A decoder-only model with 42 million parameters, tested in both autoregressive (token-by-token) and prompt (parallel) modes.
- MobileBERT: An encoder-only model used for natural language understanding tasks.
The evaluation utilized GVSoC, an accurate event-driven simulator for RISC-V architectures, coupled with analytical power models to estimate energy consumption.
Performance Analysis and Key Results
The most striking finding is the achievement of super-linear speedup. For TinyLlama in autoregressive mode, an 8-chip system provided a 26.1× speedup compared to a single-chip baseline.
As illustrated in the runtime breakdown above, the super-linear effect occurs because partitioning allows the model slices to fit entirely within the fast L2 on-chip memory. This eliminates the massive "latency tax" associated with DMA transfers from off-chip L3 memory (the light blue bars in the figure).
Key results include:
- TinyLlama (Autoregressive): 26.1× speedup on 8 chips, achieving 0.54 ms latency and 0.64 mJ energy per inference.
- TinyLlama (Prompt): 9.9× speedup on 8 chips, demonstrating better-than-linear scaling even in compute-bound scenarios.
- MobileBERT: 4.7× speedup on 4 chips without increasing total energy consumption.
- Scalability: In a scaled-up simulation with 64 MCUs, the system achieved a 60.1× speedup and a 1.3× reduction in energy for TinyLlama.
Conclusion and Future Work
While the results are promising, the study notes that as the number of chips increases, the individual matrix multiplication (GEMM) kernels become smaller, leading to lower RISC-V core utilization and diminishing returns. Future research will focus on physical multi-chip implementations and the integration of specialized hardware accelerators, such as N-EUREKA, to further drive down energy and latency for real-time edge AI applications.
Summary updated