TL;DR
Starscream is a hierarchical sharding framework designed for chiplet-based GPUs like the AMD Instinct MI300X. By introducing "orthogonal sharding" at the intra-GPU level, it overcomes the performance degradation caused by naive partitioning, delivering up to 4x speedups for attention kernels and 1.43x for fused GEMMs.
What Problem Does This Paper Address?
As Large Language Models (LLMs) grow, hardware vendors are moving away from monolithic dies toward chiplet-based architectures to maintain high manufacturing yields and compute density. The AMD Instinct MI300X, for example, integrates eight Accelerated Compute Dies (XCDs) on a single package. These GPUs offer "partitioning" modes that turn one physical GPU into multiple virtual devices with their own local memory domains (NUMA).
While partitioning promises lower kernel launch latency and better locality, current LLM frameworks (like vLLM or Megatron-LM) are "chiplet-oblivious." They treat these virtual GPUs as just "more GPUs," extending standard one-dimensional sharding strategies deeper. This leads to three critical bottlenecks:
- Memory Redundancy: Replicating KV caches or parameters across virtual GPUs wastes precious memory capacity and bandwidth.
- Arithmetic Intensity Collapse: Sharding matrices too thin makes them "skinny," turning compute-bound operations into inefficient memory-bound ones.
- Communication Overhead: Standard "flat" collective operations ignore the massive 10x bandwidth difference between internal on-package links and external inter-GPU links.
Methods
The authors propose Starscream, a rethinking of sharding grounded in the Orthogonal Sharding Principle. Instead of stretching a single sharding dimension thinner, Starscream retains the ideal sharding for the physical GPU and adds a second, orthogonal dimension for the virtual GPUs within the chip.
Key technical components include:
- Sequence-Length Sharding for Attention: Instead of replicating KV heads across XCDs (which happens in naive sharding), Starscream shards the KV cache along the sequence dimension. This eliminates redundancy and ensures each KV entry is read only once per physical GPU.
- 2D GEMM Sharding: Starscream uses "Collective Interchange." If the inter-GPU strategy shards along the row dimension ($K$), the intra-GPU strategy shards along the column dimension ($N$). This keeps matrix tiles square, preserving high arithmetic intensity.
- Hierarchical Collective Primitives: Starscream replaces flat all-reduces with a three-phase operation: an intra-GPU reduce-scatter (using fast internal links), an inter-GPU all-reduce (over external links), and an intra-GPU all-gather.
- Computation-Communication Fusion: By using symmetric memory, Starscream allows kernels to write outputs directly to the target NUMA region of the next kernel, overlapping data movement with computation and eliminating discrete collective kernel overhead.
Datasets
The paper evaluates performance using representative model shapes from state-of-the-art LLMs, including:
- Llama3-70B
- Llama3-405B
- GPT-OSS-120B
Evaluation was conducted on AMD Instinct MI300X and MI355X GPUs, testing context lengths from 16K to 128K tokens and batch sizes ranging from 1 to over 1024.
Key Insights and Ideas
- Partitioning is a Hierarchy, Not Just Scale: Treating virtual GPUs as independent devices is a mistake. The software must respect the physical hierarchy of the chiplet design to unlock its benefits.
- Orthogonal is Efficient: The "aha moment" is that switching sharding dimensions at the chiplet level prevents the "skinny matrix" problem. By keeping tiles square, the hardware stays in its compute-bound sweet spot.
- Link Heterogeneity is a First-Class Concern: In a chiplet system, not all wires are equal. Hierarchical collectives that prioritize the 10x faster internal USR links prevent the slower external XGMI links from becoming a global bottleneck.
Key Results
- Attention Speedup: Starscream's NUMA-aware paged-attention kernels achieved up to 4x speedup in partitioned modes compared to chiplet-oblivious baselines.
- GEMM Performance: Fused GEMM kernels with inter-XCD peer-mapping delivered 1.43x faster execution.
- Memory Efficiency: Starscream eliminated the KV cache replication that caused naive sharding to hit Out-of-Memory (OOM) errors at large batch sizes.
- Bandwidth Utilization: At small batch sizes, Starscream eliminated memory access "hotspotting," saturating available HBM bandwidth where standard modes failed.
Limitations and Future Work
- Tuning Overhead: The authors noted that their results are conservative lower bounds; they did not fully tune workgroup distributions or flash-attention segmentation for the specific CPX configurations.
- Extreme Scaling: At very large batch sizes (>2048), gains diminish as workloads exceed the L2 cache capacity per XCD, suggesting that future work might need to explore lower precision or even larger cache hierarchies.
- Generalization: While this study focused on Tensor Parallelism, the authors suggest these orthogonal principles should be extended to other strategies like FSDP and Pipeline Parallelism.
Summary updated