Long videos and multi-image prompts turn a multimodal language model's prompt into tens of thousands of image-patch tokens. Two separate lines of work respond to this. One makes the prompt smaller by deleting redundant visual tokens partway through the network. The other spreads the prompt across several GPUs so the first answer token arrives sooner. They interfere. Splitting a causal prompt across GPUs only works well when the split accounts for the triangular shape of causal attention, where each token looks only at tokens before it, so late tokens cost more than early ones. Pruning tokens at intermediate layers reshapes that triangle unpredictably, so a split tuned for the dense prompt becomes badly unbalanced a few layers later.
This note proposes to stop treating pruning as a disturbance to the load balancer and use it as the balancing mechanism. The idea rests on a simple fact about causal attention. Removing any one token from a prompt of $N$ tokens saves about the same total attention work, roughly $N$ query–key pairs, wherever the token sits. Where the token sits only changes which GPU gets the savings. Redundancy-based pruning also has slack: a long video contains many near-identical patches, and when the question comes at the end of the prompt, it matters little which copy is dropped. We can therefore choose which duplicates to drop per GPU and per layer so that every GPU finishes each layer at the same time, without moving any activations between devices.
Core Claim
The setting is prompt prefill (the forward pass over the whole prompt before generation) for long multimodal prompts. The prompt is split into contiguous chunks across $P$ GPUs, in the style of the relay introduced by KV-Runahead. There, each GPU computes its chunk and forwards its key–value cache (the stored attention keys and values of earlier tokens) point-to-point to the next GPU. Visual tokens are removed progressively at a few layers using a duplication criterion that needs no attention maps. The claim is that per-GPU pruning quotas, chosen from an analytic cost model, can keep the maximum-to-mean per-GPU load near 1 at every layer. This should match the accuracy of position-oblivious pruning at the same global token budget. Against both "prune and ignore the imbalance" and "prune and then migrate tokens to rebalance," it should give lower time-to-first-token (TTFT), with the largest gains when inter-GPU bandwidth is limited.
Concretely, let $n_r$ be the number of surviving tokens on GPU $r$ at a given layer and $S_r=\sum_{q<r} n_q$ the number of earlier tokens whose keys GPU $r$ must attend to and receive over the relay. The per-layer cost of GPU $r$ is then approximately
$$ T_r \approx \alpha\,\big(n_r S_r + \tfrac{1}{2}n_r^2\big) + \beta\, n_r + \gamma\, S_r , $$
where $\alpha$ is the cost per attention pair, $\beta$ the per-token cost of projections and the feed-forward block, and $\gamma$ the per-token cost of receiving relayed keys and values. The coefficients are fit once per hardware setup, replacing KV-Runahead's offline grid search with the FLOP-counting approach of Seq1F1B.
At each pruning layer we choose how many tokens each GPU keeps, the $n_r$, to minimize $\max_r T_r$. Two constraints apply: the total kept must equal the layer's budget $K$, and each GPU may drop only tokens whose duplication score exceeds a redundancy threshold. The cost is monotone in each $n_r$ and $P$ is small, so a greedy water-filling pass solves this in microseconds.
The early/late asymmetry is what makes the lever useful. Dropping a token in an early chunk shrinks the prefix $S$ for every later GPU, and also shrinks the relay traffic. Dropping a late token relieves only the last GPU.
Relation to Prior Work
The causal-imbalance literature supplies the cost model but assumes a fixed token sequence. TeraPipe uses dynamic programming to choose uneven token slices for pipeline stages. Seq1F1B replaces profiling with exact FLOP counts of each mask slice. KV-Runahead applies uneven slicing to inference prefill, reporting a 1.42x TTFT speedup over tensor/sequence parallelism on Llama 7B at 12k context on 4 GPUs. Its partitions come from a lookup table built offline for fixed text lengths. None of these systems prunes tokens, so none has to handle a triangle whose shape changes between layers. We reuse their analytic view but make the partition adaptive through pruning rather than through slicing.
Cornstarch is the closest multimodal balancer. It balances irregular multimodal attention masks across GPUs with a longest-processing-time-first assignment, and within a GPU by sharding heavy blocks. It works for training on dense visual tokens, and the authors note that below about 32k tokens its communication overhead lets simpler schemes match it. That observation constrains our design: any rebalancing that moves data at every pruning layer risks the same fate, which is why ours moves none.
Visual token reduction supplies the pruning method and the accuracy evidence. DART shows that dropping tokens similar to a small set of pivot tokens beats attention-score importance, partly because it stays compatible with fused FlashAttention kernels. We adopt its criterion for that reason. Three results argue for spreading pruning across layers rather than doing it once: FEATHER finds that one-shot early pruning induces spatial bias, ATP-LLaVA finds that shallow layers are fragile, and VisionTrim warns of text-agnostic drift. Layer-wise pruning is exactly what breaks static partitions. LOOK-M motivates the "which duplicate" freedom: text tokens receive dense attention while visual tokens receive sparse, localized attention, so exact visual positions matter less than content. All of this work runs on one GPU and ignores how pruning redistributes load across devices.
"Intra-layer parallelism" here means splitting a single layer's attention across GPUs along the sequence, in the taxonomy of UniAP, which assumes static per-layer costs. We deliberately do not use depth concurrency of the kind in CQIL and Transformer Layers as Painters. Those methods feed adjacent layers identical inputs. That rules out pruning between them, and it adds hardware work rather than removing FLOPs.
Research Program
The first step is measurement. On long-video and multi-image prompts, we run relay prefill with DART-style progressive pruning at three or four layers under the dense-optimal partition. At each layer we record per-GPU busy and idle time and the max/mean load ratio, testing whether the imbalance described in the file's Gap 1 actually occurs and how large it gets.
The second step builds the balance-aware quota solver. We fit $\alpha, \beta, \gamma$ per interconnect and integrate the solver into a relay prefill using variable-length FlashAttention with local compaction of surviving tokens. Pivot selection needs global information, so pivots (under 2% of tokens) are shared in one small broadcast per pruning layer.
The third step analyzes accuracy sensitivity. The position-freedom argument assumes the question text sits at the end of the prompt. We test it directly by placing the question before or interleaved with the visual content, measuring when position-constrained pruning starts to cost accuracy, and relaxing the redundancy threshold accordingly.
Baselines
The naive reference is dense prefill with uniform contiguous splitting under tensor/sequence parallelism, the configuration KV-Runahead beat by 1.42x on text. Next comes the dense relay with a partition tuned for the dense prompt. This is KV-Runahead's method reimplemented for the multimodal model, with no pruning. There are no Lacuna results for relay prefill of multimodal models, so we establish this number ourselves.
The central naive-pruning baseline adds DART-style progressive pruning at the same layers and global budgets as our method but keeps the dense partition. It isolates the cost of ignoring the imbalance. The accuracy side of this baseline is anchored to reported single-GPU pruning results. DART retains 98.8% of accuracy on LLaVA-1.5-7B with 88.9% of vision tokens removed and reports a 2.99x prefill speedup. VisionTrim retains 98.0% on Video-LLaVA-7B at a 93.4% pruning ratio.
The strongest competitor is migration-based rebalancing. It uses the same pruned token set, and after each pruning layer it recomputes the optimal contiguous split analytically and moves surviving activations between neighboring GPUs to realize it. This is Cornstarch's idea transplanted to inference. It is strongest because it optimizes balance exactly, with no accuracy cost from constrained token selection.
A second strong competitor stripes fixed-size token blocks round-robin across GPUs, runs ring-style context parallelism, and compacts locally after pruning. Striping averages pruning variation statistically instead of controlling it.
Two ablations isolate the key ingredient. The first keeps our per-GPU quotas but, within each GPU, drops a random above-threshold subset instead of the most redundant. This tests whether quotas cost accuracy. The second removes the communication term $\gamma$. This tests whether modeling relay traffic matters.
All methods share model weights, pruning criterion, per-layer budgets, kernels, GPU count, and interconnect, and every tunable threshold gets the same small grid search on a held-out split. Beating a baseline means lower TTFT at statistically indistinguishable accuracy (within one point on each benchmark), or higher accuracy at equal TTFT. Against migration and striping we require wins at both 10 GB/s-class and NVLink-class bandwidth to claim generality. If we win at only one of them, we will report which.
Why not just…?
Why not just prune everything before the language model, then partition the shorter prompt once? If all pruning happens at or before the first decoder layer, the sequence is known up front and a Seq1F1B-style analytic partition solves balance exactly, making this proposal unnecessary. The evidence against single-shot early pruning is indirect. FEATHER reports spatial bias from early pruning, ATP-LLaVA reports shallow-layer fragility, and on the text-KV analogue Adaptive Layer Selection reports 41.8% versus FastKV's 36.2% on InfiniteBench from choosing the pruning layer adaptively. This is a genuine threat, so we run the pre-model-only variant at matched total FLOPs on Video-MME and MileBench. If it matches progressive pruning in accuracy, we will report that the simple approach wins.
Why not just migrate tokens after each prune? This is the exact-balance solution, and it may win on fast interconnects with few pruning layers. Our bet is that per-layer migration traffic and its synchronization points hurt most at low bandwidth. The asynchronous relay's robustness there (2.7% versus 11.8% degradation under jitter in KV-Runahead) is what makes relay attractive in the first place, and Cornstarch's short-context communication overhead points the same way. The bandwidth sweep in our experiments settles this directly.
Why not just stripe blocks across GPUs so pruning averages out? With many blocks per GPU, statistical averaging may leave only a few percent imbalance, which could make the quota solver redundant. Striping gives up the contiguous prefix structure. That structure is what lets early-chunk pruning shrink relay traffic, and the relay is the source of the point-to-point communication pattern. Whether that advantage survives in practice is an empirical question. The striping baseline is included for exactly this reason, and a tie here would narrow our contribution to low-bandwidth settings.
Why not just run layers concurrently, CQIL-style, instead of splitting the sequence? CQIL reports 48.3% lower latency on LLaMA-33B, with average accuracy falling from 71.7% to 69.3%. However, it removes no FLOPs, requires identical input shapes that progressive pruning destroys, and Painters shows that order-sensitive reasoning degrades under layer parallelism. For prefill of long prompts, where GPUs are already saturated by visual tokens, removing work beats duplicating it. We therefore treat depth concurrency as orthogonal and do not evaluate it.
Why not just use one GPU once 90% of tokens are gone? For single images this is right, and we do not target that case. For hour-scale video, 10% of the visual tokens can still be tens of thousands. We include a single-GPU pruned baseline and report the prompt length above which multi-GPU prefill starts to pay off.
Decisive Experiments
The models are the Qwen2-VL family used by SpaceServe and Video-LLaVA-7B. The benchmarks are Video-MME for long video and MileBench for multi-image long context, including its needle-in-a-haystack split, where LOOK-M reports 11.7 against a full-cache 12.0 on LLaVA-1.5-13B. MMMU-Pro serves as a high-resolution sanity check. The metrics are TTFT, per-layer max/mean GPU load, relay bytes, and task accuracy, run on 4 and 8 GPUs at two interconnect bandwidths.
Three results would decide the proposal. First, whether the naive-pruning baseline shows a large, growing load ratio across pruning layers. If it does not, the problem is not real. Second, whether balance-aware quotas bring that ratio near 1 while the random-within-quota ablation shows no accuracy loss relative to position-oblivious pruning, which confirms the redundancy slack. Third, whether we beat both migration and striping on TTFT at low bandwidth. Failing the third while passing the first two would still yield a useful result: the balancing problem is real, but moving data solves it well enough.
References
- KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
- Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
- TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
- Efficient Distributed MLLM Training with Cornstarch
- Stop Looking for "Important Tokens" in Multimodal Language Models: Duplication Matters More
- Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models
- VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration
- LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference
- UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programming
- CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers
- Transformer Layers as Painters
- Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference
- SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs