Hospitals, law firms and plant operators that must keep data in-house often own a jumble of hardware: a 16GB T4 here, a 24GB RTX 3090 there, maybe a 4090, all joined by ordinary Gigabit Ethernet. A 70-billion-parameter model compressed to 4 bits does not fit on any one of these cards. Splitting the model's layers across them in sequence, so that each card runs a contiguous block and passes activations to the next (pipeline parallelism), is the natural fix. During token-by-token generation, however, such a pipeline leaves most cards idle. One remedy is self-speculative decoding. The first few layers of the model, followed by a small "early-exit" prediction head, guess the next token. The rest of the model checks that guess while the first stage is already guessing the following token. This removes the idle time without loading a second model.
Our claim is that on mixed hardware, the place where the pipeline is cut and the place where the model drafts are the same decision, and they should be optimized together. Where the first stage ends sets how deep the draft is. Draft depth sets how often guesses are accepted. The slowest card and the slowest network link set how long each step takes. Standard partitioners balance the work per card and ignore guess quality. Standard early-exit methods fix the exit layer in advance and assume identical GPUs. We propose a planner that trains exit heads at many depths once, profiles the actual cluster, and jointly picks the stage boundaries and the drafting depth to maximize accepted tokens per second under each card's memory limit, including its share of the cache of past attention keys and values (the KV cache).
Core Claim and Prior Work
The method closest to ours is PPSD. It maps an LLM onto a pipeline whose first stage drafts through an early-exit head, trained by self-distillation against the full model. It uses a "verify-while-draft" schedule so drafting and checking overlap. It reports 2.01× to 3.81× over plain autoregressive decoding, and 2.77× on XSum with LLaMA-2-13B, where naive early-exit drafting slows generation to 0.92×. PPSD supplies our drafting and scheduling mechanism. It was evaluated on homogeneous GPUs, though, with a static exit layer, and without 4-bit quantization. On a mixed cluster, its static choice interacts badly with load balancing. The cut that balances compute may put the exit too shallow, so guesses are rejected and the pipeline wastes work. Or it may put the exit too deep, so the first stage needs more memory than its card has.
EdgePipe supplies the other half. It partitions Vision Transformers across heterogeneous edge boards by dynamic programming over compute speed, memory limits and link bandwidth, and reports up to 4.16× the throughput of PipeDream. Its objective is a single forward pass, with no KV cache and no notion of guess acceptance. We extend its search so that the quantity being optimized depends on the drafting depth.
Two other systems are relevant. PipeInfer and SpecPipe already show that speculative pipelining pays off over Ethernet. Both rely on a separate draft model. That costs memory on small cards, but it is also the most serious competitor to our approach, as discussed below.
Concretely, let $P$ be an assignment of contiguous layer blocks to devices and $k$ the layer at which the draft head reads out. The first stage must end at or after $k$. Let $T(P,k)$ be the measured duration of one pipeline step, which is the maximum over stages of compute time plus transfer time. Let $A(k)$ be the expected number of tokens committed per verification when drafting from layer $k$, estimated on calibration prompts. The planner solves
$$\max_{P,k}\; \frac{A(k)}{T(P,k)} \quad \text{subject to}\quad W_s(P)+C_s(P,L) \le M_s \text{ for every device } s,$$
where $W_s$ is the 4-bit weight memory on device $s$, $C_s$ is its KV-cache memory at the target context length $L$, and $M_s$ is its usable memory. The ratio form follows directly from the verify-while-draft schedule: in steady state, each step commits $A(k)$ tokens on average and takes as long as the slowest stage. Because $A$ depends only on $k$, the search can enumerate $k$ and run an EdgePipe-style dynamic program over the remaining layers for each value. That keeps it fast enough to re-plan whenever a card is added or swapped.
Research Program
We will first train exit heads at every fourth layer of LLaMA-2-13B and a 70B model. The heads are trained by self-distillation against the AWQ 4-bit model that will actually be deployed, not against the fp16 model. This matters because CMedBench reports that 4-bit AWQ keeps a Compression Performance Score of 0.99 on clinical tasks. Nobody has measured whether quantization also changes how predictable intermediate layers are, and so how often early guesses are accepted. We will measure the acceptance curve $A(k)$ under fp16 and under AWQ.
Second, we will build the planner with a profiler for per-layer time and link bandwidth. It will also model per-stage KV memory, using a PipeInfer-style separate buffer for speculative tokens so that rejected guesses do not overwrite committed entries.
Third, we will run the system on a physical cluster of mixed T4, RTX 3090 and RTX 4090 cards over Gigabit Ethernet. This is the hardware profile the file describes as typical of on-premise deployment.
Baselines
The do-nothing reference is plain autoregressive decoding of the 4-bit model with layers split by memory alone. For 13B, the reference is instead single-GPU decoding on a 4090, since the model fits there and Laboratory-Scale AI shows single-card 4-bit deployment is the practical default. If we cannot beat that number at 13B, we will report it and restrict our claims to models that do not fit on one card.
The second baseline is standard pipeline parallelism, with stages balanced by the EdgePipe dynamic program and no speculation. This isolates the value of speculation itself.
The strongest structural baseline is PPSD applied as published. Its exit sits at a fixed layer, and the stage boundaries come from compute balancing. That is the "just compose PPSD with EdgePipe" system. On homogeneous GPUs PPSD reports 2.01× to 3.81× over autoregressive decoding. We found no result on Lacuna for PPSD on heterogeneous consumer GPUs over Ethernet. Beating it means a statistically clear gain in accepted tokens per second on the same cluster, the same heads and the same prompts.
The strongest competing method is PipeInfer with a small separate draft model, such as TinyLlama-1.1B in 4-bit, placed on whichever card the same planner chooses. PipeInfer reports 1.5× to 2.15× over standard speculative inference.
All systems will share the same quantized weights, prompts and context lengths. Each gets the same profiling budget. The draft model and the exit heads get equal distillation compute on the same data.
The key ablation compares three planners: the joint planner, the balanced-then-exit composition, and an exhaustive oracle over all $(P,k)$ pairs. The gap between the composition and the joint planner is the contribution. The gap between the joint planner and the oracle measures how well the acceptance estimate holds up.
Why not just…?
Why not just run a 4-bit model on the biggest single GPU? For 13B on a 24GB card, this may well win, and we will say so if it does. The file notes, however, that 4-bit models generate slowly on low-end hardware because generation is limited by memory bandwidth. A 4-bit 70B model also simply exceeds 24GB. The proposal's regime is models that must be split. The 13B single-card comparison is included precisely to mark where that regime begins.
Why not just use a tiny separate draft model, as PipeInfer and SpecPipe do? This is the real threat. A 1.1B draft model costs well under a gigabyte in 4 bits, and it may be accepted more often than a shallow exit head. Self-drafting has two possible advantages. It needs no second vocabulary-aligned model. And it reuses the first stage's computation, so verification starts partway through the network rather than from scratch. Whether these advantages survive on mixed cards is an empirical question that the head-to-head comparison settles. If the draft model wins on accepted tokens per second at equal memory, we would retarget the planner to choose the draft model's placement instead. The joint-planning idea transfers, but the self-speculative framing would not.
Why not just split matrices across GPUs with tensor parallelism? Tensor parallelism synchronizes the cards inside every layer. SpecPipe and PipeInfer both motivate pipeline splitting by the fact that this per-layer synchronization needs NVLink-class links rather than Ethernet. We will run vLLM-style tensor parallelism on two matched cards over Ethernet once, as a sanity check, rather than as a main baseline.
Why not just compose PPSD with EdgePipe's partitioner? That composition is exactly our main ablation. It becomes equivalent to our method if the acceptance curve is nearly flat across candidate exit depths. In that case, the balanced cut is already near-optimal, and our planner adds nothing.
Decisive Checks
We will evaluate on XSum, GSM8K and HumanEval, following PPSD. We add MedQA from CMedBench as a clinical workload. With greedy decoding, speculation must leave outputs bit-identical to the non-speculative model, so MedQA accuracy should match the non-speculative 4-bit model exactly. We verify this rather than assume it.
The idea stands or falls on three measurements. First, $A(k)$ must vary enough across depths that the joint planner beats the balanced composition by a clear margin on at least two of the three cluster configurations. If it does not, we report the flat curve as a negative result. Second, the joint planner must land close to the exhaustive oracle, which shows the profiled estimate is trustworthy. Third, at 70B the method must match or beat PipeInfer with a separate draft model at equal memory. If quantization flattens or shifts the acceptance curve relative to fp16, that is itself a finding about where 4-bit models can draft from.
References
- Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding
- Pipeline Parallelism for Inference on Heterogeneous Edge Computing
- PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
- SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding
- CMedBench: A Comprehensive Benchmark for Efficient Medical Large Language Models
- Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings