Starscream: Reimagining LLM Sharding on AMD Instinct GPUs with GPU Partitioning

Sumanth Gudaparthi, Smeet Chheda, Ganesh DasikaarXiv 2026 · arXiv

TL;DR

Starscream is a hierarchical sharding framework designed for chiplet-based GPUs like the AMD Instinct MI300X. By introducing "orthogonal sharding" at the intra-GPU level, it overcomes the performance degradation caused by naive partitioning, delivering up to 4x speedups for attention kernels and 1.43x for fused GEMMs.

What Problem Does This Paper Address?

As Large Language Models (LLMs) grow, hardware vendors are moving away from monolithic dies toward chiplet-based architectures to maintain high manufacturing yields and compute density. The AMD Instinct MI300X, for example, integrates eight Accelerated Compute Dies (XCDs) on a single package. These GPUs offer "partitioning" modes that turn one physical GPU into multiple virtual devices with their own local memory domains (NUMA).

While partitioning promises lower kernel launch latency and better locality, current LLM frameworks (like vLLM or Megatron-LM) are "chiplet-oblivious." They treat these virtual GPUs as just "more GPUs," extending standard one-dimensional sharding strategies deeper. This leads to three critical bottlenecks:

  1. Memory Redundancy: Replicating KV caches or parameters across virtual GPUs wastes precious memory capacity and bandwidth.
  2. Arithmetic Intensity Collapse: Sharding matrices too thin makes them "skinny," turning compute-bound operations into inefficient memory-bound ones.
  3. Communication Overhead: Standard "flat" collective operations ignore the massive 10x bandwidth difference between internal on-package links and external inter-GPU links.

Methods

The authors propose Starscream, a rethinking of sharding grounded in the Orthogonal Sharding Principle. Instead of stretching a single sharding dimension thinner, Starscream retains the ideal sharding for the physical GPU and adds a second, orthogonal dimension for the virtual GPUs within the chip.

Key technical components include:

  • Sequence-Length Sharding for Attention: Instead of replicating KV heads across XCDs (which happens in naive sharding), Starscream shards the KV cache along the sequence dimension. This eliminates redundancy and ensures each KV entry is read only once per physical GPU.
  • 2D GEMM Sharding: Starscream uses "Collective Interchange." If the inter-GPU strategy shards along the row dimension ($K$), the intra-GPU strategy shards along the column dimension ($N$). This keeps matrix tiles square, preserving high arithmetic intensity.
  • Hierarchical Collective Primitives: Starscream replaces flat all-reduces with a three-phase operation: an intra-GPU reduce-scatter (using fast internal links), an inter-GPU all-reduce (over external links), and an intra-GPU all-gather.
  • Computation-Communication Fusion: By using symmetric memory, Starscream allows kernels to write outputs directly to the target NUMA region of the next kernel, overlapping data movement with computation and eliminating discrete collective kernel overhead.
The AMD Instinct MI300X architecture (A) and the speedup achieved by Starscream vs. naive sharding (B).

Datasets

The paper evaluates performance using representative model shapes from state-of-the-art LLMs, including:

  • Llama3-70B
  • Llama3-405B
  • GPT-OSS-120B

Evaluation was conducted on AMD Instinct MI300X and MI355X GPUs, testing context lengths from 16K to 128K tokens and batch sizes ranging from 1 to over 1024.

Key Insights and Ideas

  • Partitioning is a Hierarchy, Not Just Scale: Treating virtual GPUs as independent devices is a mistake. The software must respect the physical hierarchy of the chiplet design to unlock its benefits.
  • Orthogonal is Efficient: The "aha moment" is that switching sharding dimensions at the chiplet level prevents the "skinny matrix" problem. By keeping tiles square, the hardware stays in its compute-bound sweet spot.
  • Link Heterogeneity is a First-Class Concern: In a chiplet system, not all wires are equal. Hierarchical collectives that prioritize the 10x faster internal USR links prevent the slower external XGMI links from becoming a global bottleneck.
Illustrative GPU sharding for GQA (A) and RowParallelLinear (B), showing how naive sharding leads to redundancy or skinny matrices.

Key Results

  • Attention Speedup: Starscream's NUMA-aware paged-attention kernels achieved up to 4x speedup in partitioned modes compared to chiplet-oblivious baselines.
  • GEMM Performance: Fused GEMM kernels with inter-XCD peer-mapping delivered 1.43x faster execution.
  • Memory Efficiency: Starscream eliminated the KV cache replication that caused naive sharding to hit Out-of-Memory (OOM) errors at large batch sizes.
  • Bandwidth Utilization: At small batch sizes, Starscream eliminated memory access "hotspotting," saturating available HBM bandwidth where standard modes failed.

Limitations and Future Work

  • Tuning Overhead: The authors noted that their results are conservative lower bounds; they did not fully tune workgroup distributions or flash-attention segmentation for the specific CPX configurations.
  • Extreme Scaling: At very large batch sizes (>2048), gains diminish as workloads exceed the L2 cache capacity per XCD, suggesting that future work might need to explore lower precision or even larger cache hierarchies.
  • Generalization: While this study focused on Tensor Parallelism, the authors suggest these orthogonal principles should be extended to other strategies like FSDP and Pipeline Parallelism.

Summary updated

Abstract

As GPU accelerators grow larger to meet rising compute demands, they exhibit increasingly pronounced non-uniform memory access (NUMA) characteristics that challenge existing LLM frameworks. Chiplet-based architectures provide a practical path to larger GPUs while mitigating manufacturing yield constraints. Modern GPUs such as AMD Instinct MI300X employs such a multi-die design, and further offer GPU partitioning capabilities, partitioning a single physical GPU into multiple logical devices with NUMA-aware memory domains, promising faster kernel launches, and reduced memory access latencies with improved locality. However, current parallelization strategies, designed for homogeneous GPU topologies, when applied under partitioning, extend its unidimensional sharding causing significant performance degradation. Three bottlenecks emerge: (1) memory redundancy; (2) arithmetic intensity collapse; (3) communication overhead from flat collectives. We present Starscream, a rethinking of LLM sharding for the chiplet era. Rather than stretching inter-GPU sharding thinner, Starscream adds intra-GPU sharding along an orthogonal dimension, grounded in three foundational principles: kernel shape preservation, memory redundancy elimination, and communication overhead minimization. These principles form the basis for extending any sharding strategy into the partitioning regime. We demonstrate their effectiveness using tensor parallelism as a case study. On AMD MI300X and MI355X, exploiting partitioning capabilities yields up to 4x speedup for attention and 1.43x for fused GEMMs compared to chiplet-oblivious baselines, demonstrating the effectiveness of chiplet-aware designs. Starscream's principles generalize across sharding strategies and future architectures.
BibTeX
@misc{gudaparthi2026starscream,
  author = {Sumanth Gudaparthi and Smeet Chheda and Ganesh Dasika},
  title = {{Starscream: Reimagining LLM Sharding on AMD Instinct GPUs with GPU Partitioning}},
  year = {2026},
  eprint = {2611.15951},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15951},
}
Figure