Spend Redundancy Only Where the Pipeline Waits

When a large language model is too big for one GPU, it is often split into consecutive chunks of layers, one chunk per GPU, and requests stream through them like an assembly line. This is pipeline parallelism. Its steady-state throughput is set entirely by the slowest chunk: every other GPU finishes early and sits idle. Separately, compression research shows that large models carry a great deal of spare capacity. Whole late transformer blocks can be replaced by a single linear map, and half the weights can be removed in one pass, with small accuracy loss. Today these two facts are used independently. Compression is applied uniformly across the model, and the pipeline is partitioned afterwards as if every layer's cost were fixed. Both habits waste something. Compressing a layer on a GPU that already sits idle buys no throughput, yet it still costs accuracy.

This note proposes treating each layer as a job whose run time can be shortened, at a price in accuracy, and solving partitioning and compression as one problem. The goal is the least total accuracy damage for which the slowest stage meets a target latency. The claim is conditional, and we mean it that way. When the partitioner cannot balance the stages on its own, spending compression only on the bottleneck should beat compress-then-partition by a margin that grows with that leftover imbalance. When it can balance them, the two approaches should coincide. Imbalance comes from coarse layer granularity, a huge vocabulary output head, mixture-of-experts or hybrid layers, memory limits, or mixed GPU types. Because the claim is conditional, the proposal can be falsified.

Core Claim

Scheduling theory says that finishing time is bounded by the longest dependency chain. The hybrid-platform approximation work studies how to trade that critical path against keeping machines busy. For a pipeline running at steady state, the equivalent quantity is the bottleneck stage latency, which is the objective that SliceGraph's pipeline partitioning work minimizes over a DNN's computation graph. SliceGraph and AdaPtis, which co-searches partition, GPU placement and micro-batch order for heterogeneous models, both treat each operator's latency as fixed. That is the gap this proposal targets.

Three compression papers supply per-layer options with measured accuracy costs. ReplaceMe swaps redundant contiguous blocks for a fused linear map without fine-tuning. SparseGPT prunes in one shot, including hardware-friendly 2:4 patterns in which two of every four weights are zero. GPTailor finds that redundancy concentrates in late layers.

Formally, let layer $\ell$ choose an option $o_\ell$ from its menu (keep, 2:4 sparsify, linearize), with latency $t_\ell(o_\ell)$ and a calibration-measured cost $c_\ell(o_\ell)$. The pipeline cuts the layer sequence into $k$ contiguous stages $S_1,\dots,S_k$ on devices with speed $s_i$ and memory $M_i$. We solve

$$\min_{\{o_\ell\},\{S_i\}} \sum_\ell c_\ell(o_\ell) \quad \text{s.t.} \quad \max_i \frac{1}{s_i}\sum_{\ell\in S_i} t_\ell(o_\ell) \le T,\;\; \mathrm{mem}(S_i)\le M_i,$$

and sweep the target latency $T$ to trace a curve of throughput against accuracy.

Transformers are nearly a linear chain of layers. For a fixed $T$ this is therefore an exact dynamic program over (layer index, stages used), with a knapsack-style inner choice of options. It is a direct extension of SliceGraph's dynamic-programming step, and it needs no genetic search over orderings.

The sharp prediction concerns what the joint solver does that uniform compression cannot. On Gemma-style models the last stage carries a very large vocabulary head and is the bottleneck. The joint solver should concentrate linearization on the late blocks that share that stage, which are exactly the blocks GPTailor and ReplaceMe identify as most redundant. It should leave the early layers alone, even though those early layers may sit on idle GPUs.

Research Program

We first build per-layer menus for each model. This means profiling the latency of every option on the target GPUs, and computing the costs on a shared calibration set. The linearization cost is ReplaceMe's cosine distance. The sparsification cost is SparseGPT's reconstruction error.

The additive cost is only a proxy, since errors compound across layers. A second step therefore takes the dynamic program's top few solutions, evaluates their true downstream accuracy, and runs a small local search that swaps one option at a time. We report separately how often the proxy's ranking holds up.

To handle more than a linear chain, we extend the program in three ways. Mixture-of-experts layers get expert-dropping options, borrowing the redundancy view of MoNE. Hybrid attention/Mamba layers from the AdaPtis model set are included. Heterogeneous GPU speeds enter through the factors $s_i$.

The main analysis quantity is the imbalance residual. It is the bottleneck latency of the best compression-free partition divided by (total latency / $k$), that is, by the latency the stages would have if the work were split perfectly evenly. Our hypothesis is that the joint method's advantage over compress-then-partition rises with this residual and vanishes near 1.

A second analysis concerns model size. The file reports that one-shot pruning fails on small models (Sparse-Dense-Sparse) and that redundancy grows with scale (activation-sparsity study). We therefore expect the menus to become cheaper as models get larger. The set of feasible targets $T$ should then widen, and a fixed accuracy budget should buy more throughput.

Baselines

The naive reference point is uncompressed pipeline inference with an equal number of layers per stage, which is the default Megatron-style layout. It fixes the bubble ratio and throughput we start from.

The strongest scheduling-only competitor is the best compression-free partition. For chain-shaped models we compute it with SliceGraph's exact contiguous dynamic program. For heterogeneous models we use an AdaPtis-style profile-guided search over partition and placement. SliceGraph reports solutions within 4.5% of the proven optimum for up to 16 stages, and AdaPtis reports 1.34× average speedup on Gemma over Megatron-LM I-1F1B. That AdaPtis figure measures training throughput. We found no reported inference result on Lacuna for AdaPtis, so we re-run its search idea ourselves under our inference profiler.

The strongest compression competitor is uniform compression followed by optimal repartitioning. We run ReplaceMe at 25% depth, where it reports 92.5% of baseline accuracy on Llama-2-7B, and SparseGPT 2:4, which moves OPT-175B perplexity from 8.34 to 8.74. We also run GPTailor's configuration where one is available; GPTailor reports 97.3% retention at about 25% parameter reduction on Llama2-13B.

The key ablation is a latency-aware sequential baseline. It ranks layers by latency saved per unit of cost, compresses greedily until the budget is spent, and only then partitions. This isolates what jointness adds beyond a sensible compression ranking.

All methods share the same calibration data, the same accuracy-evaluation suite, the same GPUs and kernels, and the same profiler. The search methods get equal wall-clock budgets. Comparisons are made in two directions: throughput at matched accuracy, and accuracy at matched throughput. Beating a baseline means one of two things. Either the joint method delivers higher steady-state tokens per second at equal OpenCompass retention, or it matches throughput with retention higher than the run-to-run variance across calibration seeds.

Why not just…?

Why not just repartition better? This is the real threat. On a homogeneous model whose layer count divides evenly across stages, a good partitioner already balances the load, and our method should give nothing beyond ordinary compression. We predict exactly that and test it. The joint method's case rests on settings where partitioning leaves a residual. AdaPtis reports standard bubble ratios above 40–50% on heterogeneous models, and a vocabulary head or a mixture-of-experts layer cannot be split across contiguous stage cuts. If the gain does not track the imbalance residual, the idea fails.

Why not just compress uniformly and then partition? Once the model is repartitioned, uniform compression does help every stage. But it charges accuracy for latency savings in places where a cheaper option on a redundant late layer would have delivered the same bottleneck reduction. GPTailor's finding that early layers must be preserved while late layers can be removed implies that uniform budgets misprice the options. The latency-aware sequential ablation is the fair form of this alternative. If it matches the joint method, the gain comes from the ranking rather than from coupling compression with scheduling, and we will say so.

Why not just quantize? Quantization mostly relieves memory and bandwidth. It keeps the same sequence of layers, so it cannot remove a block from the bottleneck stage, and SparseGPT reports that pruning and 4-bit quantization compound rather than compete. We therefore treat INT4 as a base applied to every method, not as a rival. A speculative-decoding pipeline such as SpecPipe targets per-token latency rather than throughput, and is likewise orthogonal. Swapping in a smaller dense model is weakened by the Double Sparse Factorization result that a 50%-pruned 13B model beats a dense 7B model.

Proposed Experiments

The models are Llama-2-13B and Llama-3-8B, both near-homogeneous controls; a Gemma model with a large vocabulary head; and a DeepSeek-style mixture-of-experts model and a Nemotron-H hybrid, both drawn from the AdaPtis model set. Each runs on 4- and 8-stage inference pipelines, plus one mixed-GPU configuration. Accuracy is measured on GPTailor's OpenCompass task subset (MMLU, HellaSwag, PIQA, BoolQ, CommonSenseQA, Race, and others), on WikiText and C4 perplexity, and on GSM8K. GSM8K is included because the activation-sparsity study reports that reasoning tolerates far less compression. Systems metrics are steady-state throughput, bottleneck stage latency, and bubble ratio.

Three checks decide the question. First, plot the joint method's advantage over latency-aware sequential compression against the imbalance residual across all model and stage-count combinations. We expect a monotone relationship with near-zero gain on the divisible Llama controls. Second, on Gemma, confirm that the chosen compressions land in the last stage's late blocks, and that forcing the same budget onto early layers costs more accuracy for the same throughput. Third, validate the additive cost proxy by rank correlation against measured downstream retention. If that correlation is weak on GSM8K, restrict the method to options whose costs combine predictably, and report the failure.