Harvesting Expert-Parallel Bubbles for Fine-Tuning

Oana Balmau, Saurav Banka, Jiaxuan Chen, Anne-Marie Kermarrec, Ramya Narasimha Prabhu, Rafael Pires, Mathis Randl, Martijn de VosarXiv 2026 · arXiv

TL;DR

BubbleTea is a novel colocation system that harvests "bubbles"—idle time caused by load imbalance in Mixture-of-Experts (MoE) models—to perform background fine-tuning. By predicting these bubbles per-GPU and filling them with micro-chunks of training work, it achieves up to 6.2× higher training throughput than state-of-the-art systems while keeping inference latency within strict service-level objectives.

What Problem Does This Paper Address?

Serving Large Language Models (LLMs) is expensive, leading researchers to "colocate" best-effort tasks like fine-tuning alongside latency-critical inference. However, modern LLMs increasingly use Mixture-of-Experts (MoE) architectures. In an MoE model, only a subset of "experts" is active for any given token. To handle massive models, these experts are often distributed across multiple GPUs using Expert Parallelism (EP).

Because token routing is input-dependent, some GPUs inevitably receive more work than others. Lightly loaded GPUs finish their expert computation early and must sit idle—creating a "bubble"—while waiting for the most-loaded GPU to finish so the model can proceed to the next layer. Even with state-of-the-art load balancing, these bubbles consume 8–13% of the critical "prefill" time. Existing colocation systems miss these opportunities because they look for "headroom" at the level of an entire inference request or iteration, rather than the micro-scale, per-GPU gaps inside a single MoE layer.

Figure 2. Expert-parallel bubbles. Token routing assigns differ- ent amounts of expert work to different GPUs. GPU 0 finishes its local expert computation (in green) before the more heavily loaded GPU 1 but cannot proceed until the cross-GPU dependency clears. We define this waiting interval (in red) as a bubble.

Methods

BubbleTea introduces three main innovations to "harvest" these bubbles:

  1. Bubble Prediction: Since the router decides token assignments before experts run, BubbleTea uses a small GPU kernel to predict the exact duration of the upcoming bubble for every GPU at every layer.
  2. Fine-Grained Training Chunks: The authors developed a compiler that automatically derives a LoRA (Low-Rank Adaptation) backward pass from the serving engine's code. Crucially, it partitions this work into tiny "chunks" (under 50 microseconds) that can fit into the predicted bubbles.
  3. In-Place Training: Unlike systems that duplicate model weights for training, BubbleTea trains on the weights already loaded for serving. This saves significant GPU memory (e.g., 15 GB for a 30B model) that can instead be used for the KV cache.
  4. Device-Side Gating: To ensure training never delays inference, the serving worker manages a "gate" in GPU memory. It opens the gate when a bubble starts and closes it shortly before the bubble is predicted to end, ensuring no training chunk is running when the next inference step needs to begin.
Figure 4. The system architecture of BUBBLETEA. On each GPU, a serving worker and a fine-tuning worker run side by side and share the same copy of the base model weights.

Datasets

The system was evaluated using:

  • Models: Qwen3-30B-A3B, Qwen1.5-MoE-A2.7B, and DeepSeek-V2-Lite.
  • Inference Workloads: arXiv summarization prompts (1k–32k tokens) and production traces from Azure LLM (code and conversation).
  • Fine-tuning Workload: The Alpaca dataset (52,000 instruction-following prompts).

Key Insights and Ideas

  • Bubbles are predictable: The "aha moment" is realizing that the router's decision provides a perfect signal for per-GPU idle time before the idle time even happens.
  • Prefill is the goldmine: While previous systems focused on the "decode" phase (generating tokens one by one), BubbleTea finds that the "prefill" phase (processing the initial prompt) offers much larger and more useful bubbles for MoE models.
  • Per-GPU planning is mandatory: Global metrics for "system slack" are useless for EP because the system as a whole looks busy (due to the straggling GPU), even when most GPUs are idle.
  • Safety through "Early Close": By closing the admission window slightly before the predicted end of a bubble, BubbleTea absorbs prediction errors and prevents training from ever becoming the bottleneck.

Key Results

  • Throughput Gains: BubbleTea delivers fine-tuning throughput 1.9–6.2× faster than LLMStation (a state-of-the-art colocation baseline).
  • Production Performance: On bursty, real-world traces, it trains 3.3× faster than LLMStation and 14× faster than uncoordinated training (running a standard trainer alongside serving).
  • Latency Preservation: In 50 out of 63 tested workloads, BubbleTea kept the 90th-percentile inference latency within 10% of serving alone. On production traces, it stayed within 7%.
Figure 1. BUBBLETEA trains 3.3× faster than LLMSTATION and 14× faster than uncoordinated colocation on a production trace, while keeping p90 TTFT within 3% of serving alone. Load (top) and fine-tuning throughput (bottom) of BUBBLETEA and baselines over time on a QWEN3-30B-A3B model serving a production (Azure) trace at its original rate (with 30 s bins). Legend values give each system’s p90 TTFT over the whole trace, relative to serving alone.

Limitations and Future Work

  • Queue Awareness: The current policy does not pause training when inference requests are actively queuing. The authors suggest a "queue-aware cutoff" as a future improvement.
  • Memory Overhead: While it avoids weight duplication, the system still requires memory for "activation pools" to support the training chunks, which reduces the available KV cache by about 5–7%.
  • Compiler Support: Supporting entirely new model architectures requires the compiler to have differentiable equivalents for specific kernels (like specialized attention or MoE kernels).

Summary updated

Abstract

GPUs are expensive, yet latency-critical large language model (LLM) inference leaves part of them idle. Colocation reclaims this capacity by running best-effort work, such as fine-tuning, alongside serving. Many leading LLMs now use a Mixture-of-Experts (MoE) architecture, which adds a new source of idle time. Under expert parallelism, MoE experts are spread across GPUs, and because the router sends uneven numbers of tokens to each GPU, lightly loaded GPUs wait for the most loaded one. Even after applying load balancing solutions, such bubbles still take 8--13% of prefill time. We present BubbleTea, the first colocation system that leverages expert-parallel bubbles for fine-tuning. Unlike existing colocation systems, which make one scheduling decision for all GPUs per forward pass, BubbleTea predicts each GPU's bubble at every layer from the router's token assignment, and fills it with short chunks of fine-tuning work that reuse the weights already loaded for serving. We also introduce compiler passes that automatically derive these chunks from the served LLM as-is, without porting each model architecture to a separate training engine. Our evaluation on three MoE models, served on four A100 GPUs, shows that fine-tuning throughput with BubbleTea is 1.9--6.2$\times$ faster than with LLMStation, a state-of-the-art colocation system. BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads, compared to 44 for LLMStation. On bursty production traces, BubbleTea trains 3.3$\times$ faster than LLMStation and 14$\times$ faster than a stock trainer colocated under CUDA MPS, with p90 inference latency staying within 7% of serving alone.
BibTeX
@misc{balmau2026harvesting,
  author = {Oana Balmau and Saurav Banka and Jiaxuan Chen and Anne-Marie Kermarrec and Ramya Narasimha Prabhu and Rafael Pires and Mathis Randl and Martijn de Vos},
  title = {{Harvesting Expert-Parallel Bubbles for Fine-Tuning}},
  year = {2026},
  eprint = {2611.24496},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24496},
}
Figure