Split inference runs the first part of a neural network on a phone, car, or sensor and the rest on a server. The edge device only needs to send the intermediate activations, the numbers flowing between layers, but those activations are often larger than the raw input. Prior work shrinks them with fixed recipes, such as dropping high-frequency Fourier coefficients or pruning spatial tokens. These recipes only work at very shallow split points, or they require training an encoder on the device.
We propose a different approach: send activations as coordinates in a low-dimensional principal-component subspace, meaning the few directions along which the activations vary most. The server owns that subspace and keeps it current. The device computes a per-sample reconstruction error at no extra cost, and that error tells the server when the input distribution has drifted and the subspace has gone stale. When it has, the device uploads a small, uniformly sampled batch of full activations, and the server updates the basis.
The bet is that this gives two things at once. Channel-wise compression should work at deep split points, where Fourier methods fail. And it should stay accurate when deployment data looks nothing like the small calibration set used to fit the basis. Both are open problems in the material we build on.
Core Claim
Every PCA-based compressor we found fits its projection once, on a small and clean calibration set, and then freezes it:
- La RoSA builds layerwise rotations from WikiText2.
- SP³ projects hidden dimensions using calibration features.
- Loki computes key-vector PCA from calibration data.
- The dense-retrieval study by Ma et al. compresses 768-dimensional BERT vectors with PCA, and outperforms supervised projections. It also documents steeper degradation when a Wikipedia-fitted basis is applied to SQuAD.
That last observation is the motivating evidence. A frozen basis is a fixed geometric prior, and such priors are known to break when moving from small curated data to heterogeneous data. Liu et al. say this explicitly about their small-cohort distance constraints.
Meanwhile, the closest split-inference work, FourierCompress, compresses the first transformer layer's activation with a 2D Fourier transform. Its authors report that deeper splits fail because activations lose smooth structure. Our claim has two parts:
- Deep-layer compression. A data-dependent channel subspace can compress deep-layer split activations where a fixed frequency basis cannot.
- Cheap drift tracking. The subspace can be kept accurate under distribution drift at a communication cost far below re-sending raw activations. Tracking is driven by a drift signal the device gets for free.
Concretely, the device holds a basis $U \in \mathbb{R}^{D \times k}$ and a mean $\mu \in \mathbb{R}^D$, where $D$ is the channel width and $k \ll D$ is the number of retained components. For each token or spatial activation $x$, it transmits the coefficients $z = U^\top (x - \mu)$. It also computes the residual fraction
$$ r(x) = 1 - \frac{\lVert z \rVert^2}{\lVert x - \mu \rVert^2}, $$
which is the share of the activation's energy that the current basis misses. Computing $r$ costs one extra norm on top of the projection the device already performs. The server reconstructs $\hat{x} = U z + \mu$ and continues the forward pass. It also tracks a running average of $r$ across incoming traffic.
Research Program
The system has four parts. The first three are the core design; the fourth is a variant we test separately.
1. Choosing the split point. We first measure, layer by layer, how much activation variance falls into the top $k$ components for each model we study.
SP³'s success at pruning deeper hidden dimensions is suggestive, but it concerns weight-side structure. It does not establish that split activations are low-rank, so we treat that as a hypothesis to verify rather than a premise.
We also handle a known complication. Deep LLM activations contain a few massive outlier channels and a dominant mean direction. Following the purification idea of Raunak et al., which removes the mean and dominant components around a PCA projection, we:
- subtract the mean,
- send a fixed small set of outlier channels raw, and
- apply PCA to the remainder.
The retained coefficients are then quantized to 8 or 4 bits. La RoSA and GPLQ both suggest that working in a PCA-rotated coordinate system makes value distributions better behaved for pruning and quantization.
2. Refreshing the basis. When the server's running average of $r$ exceeds a threshold, it asks the device for a small batch of full activations. The device picks these uniformly at random over recent traffic, not by loss or residual. This choice is deliberate: Arazo et al. show that importance-weighted selection under a budget biases estimates and loses data variety, while uniform sampling holds up.
The server then:
- updates $U$ and $\mu$ with an incremental SVD that blends old and new statistics,
- increments a version number, and
- pushes the new basis to the device.
The push costs $Dk$ numbers. In-flight coefficients carry their basis version, so the server always reconstructs with the matching basis. Neither side needs gradient computation.
3. Accounting for communication. The total communication cost includes the coefficient stream, the uploaded refresh samples, and the basis pushes. All results report bytes per token or per image amortized over the whole stream.
4. Task-aware variant. The server already runs the downstream layers. It can therefore weight the covariance by a cheap estimate of each direction's influence on the output, using gradients computed server-side on the uploaded samples. The device still only performs a matrix multiply.
This variant tests whether task awareness can live entirely in the cloud. That would address the tension between training-free PCA and the learned, task-guided encoders of CDC and CAFC-SE. We report it as a separate variant, not as part of the core claim.
Baselines
Uncompressed reference. Activations sent at FP16 from the same split point. This fixes the accuracy ceiling and the byte count every method is measured against.
Quantization only. The same activations quantized to INT8 and INT4, using analytical clipping and per-channel ranges in the spirit of Banner et al.. This is the cheapest serious competitor, because it needs no basis at all. To beat it, our method must reach equal accuracy at fewer bytes.
FourierCompress. The strongest existing split-inference method in the file. It reports a 7.6× activation size reduction with under 0.3% accuracy loss across 10 commonsense QA benchmarks on Llama 3 and Qwen 2.5, at layer 1 (FourierCompress).
- We run it at layer 1, which is its favored setting, and at our deeper split points.
- It uses the same models, benchmarks, and quantization of retained coefficients.
- Beating it means a higher compression ratio at equal or lower accuracy loss, at a split point where it degrades. Matching it at layer 1 is not enough.
Top-K activation sparsification. The device keeps only the largest-magnitude activation values and sends them, at matched byte budgets.
Static PCA. Fitted once on WikiText2 calibration data, as in La RoSA, and never updated. This is the ablation that isolates our key ingredient: drift tracking.
Two refresh-rule ablations. These separate the effect of the drift trigger from the effect of uniform sampling. Both use the same refresh byte budget as our method.
- Fixed-schedule refresh: the basis is updated every $N$ tokens, regardless of residual.
- Hard-sample refresh: uploads the highest-residual activations instead of uniform ones.
Learned bottleneck. A CDC-style autoencoder, guided by a classifier, at the same split point (CDC). It is trained on the same calibration data with a tuning budget matched in GPU-hours to our server-side basis computation. It is also given the option of periodic on-device fine-tuning, with that compute counted.
Fairness rule. Every method is evaluated on the same frozen backbone, the same data streams, and an identical hyperparameter search budget per method.
Why not just…?
Why not just fit PCA once on a large, diverse calibration set? This is the most serious threat. If a broad calibration set spans the deployment distribution, tracking adds nothing. The static-PCA baseline is run twice to settle this: once with WikiText2, and once with a calibration pool mixing all evaluation domains' training splits. If the broad static basis matches our method on the drifting stream, we will report that drift tracking is unnecessary for these shifts. In that case the contribution shrinks to deep-layer channel compression alone. The evidence from Ma et al. that out-of-domain PCA degrades suggests the gap is real, but it does not prove the gap survives a better calibration set.
Why not just quantize activations and skip PCA? Quantization and projection compound rather than compete. Quantization reduces bits per number; PCA reduces how many numbers are sent. The open question is whether deep-layer activations have enough low-rank structure for the projection to pay for its error. Naive 4-bit activation quantization is known to be fragile: in Banner et al., ResNet101 drops from 77.2% to 62.6% at 4W4A before their corrections. That makes INT4 alone a weak way to reach high compression ratios. The quantization-only baseline at matched bytes settles the question directly.
Why not just use a fixed transform like Fourier, which needs no calibration? FourierCompress itself reports that its method fails past layer 1. Many deployments want deeper splits, so that the device does a meaningful share of the work and the server never sees near-raw input. Running FourierCompress at our deep split points shows the size of the gap. If FourierCompress at layer 1 already meets the latency and privacy needs of a deployment, our method's advantage is limited to settings that need deeper splits, and we will say so.
Why not just train a small encoder on the device? Learned bottlenecks such as CDC need tuned loss weights; CDC uses an attenuation factor of 48 for images and 16 for audio. Re-adapting them under drift requires backpropagation on the device. The supervised projections in Ma et al. also overfit compared with PCA. The learned-bottleneck baseline, with its compute counted, tests whether nonlinear capacity wins at matched bytes despite these costs.
Proposed Experiments
Language models. The primary testbed is Llama 3 (1B, 3B) and Qwen 2.5 (1.5B, 3B), split at layers 1, L/4, L/2, and 3L/4. The basis is calibrated on WikiText2. We evaluate on the 10 commonsense benchmarks used by FourierCompress, plus a drifting stream that concatenates those tasks with MATH500 and GPQA-Diamond prompts, as used in La RoSA. We measure:
- accuracy,
- amortized bytes per token,
- on-device projection latency, and
- recovery time after each domain switch, defined as the number of tokens until accuracy returns to within 0.5% of its in-domain level.
Vision. The secondary testbed is a Swin-T or ViT backbone split at a middle block. It is calibrated on ImageNet-1K and streamed onto the five fine-grained classification datasets used by GPLQ. We found no reported split-inference result on Lacuna for this setting, so the uncompressed and static-PCA runs serve as the reference.
Composition with spatial pruning. A third check applies channel PCA to the sparse features kept by T-FFC on KITTI. T-FFC reports 6061× compression with under 3% AP loss, and A-FFC reports 901× with near-zero loss (Feature Compression for Cloud-Edge Multimodal 3D Object Detection). This tests whether channel and spatial compression compound in practice.
Decisive checks. The three results that decide the proposal are:
- Variance spectra. Do layer-wise variance spectra show that deep split activations put most of their energy in a small $k$ once outlier channels are removed? If not, the premise fails.
- Deep-split comparison. At a deep split, does our method beat both FourierCompress and INT4 quantization at matched bytes?
- Drift tracking. On the drifting stream, does residual-triggered, uniformly sampled refresh beat both the broad static basis and fixed-schedule refresh, with refresh traffic counted?
References
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
- $\rm SP^3$: Enhancing Structured Pruning via PCA Projection
- Loki: Low-rank Keys for Efficient Sparse Attention
- Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval
- Inter-modality Relationship Constrained Multi-Task Feature Selection for AD/MCI Classification
- FourierCompress: Layer-Aware Spectral Activation Compression for Efficient and Accurate Collaborative LLM Inference
- Effective Dimensionality Reduction for Word Embeddings
- GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
- How Important is Importance Sampling for Deep Budgeted Training?
- CDC: Classification Driven Compression for Bandwidth Efficient Edge-Cloud Collaborative Deep Learning
- Codebook-Based Adaptive Feature Compression With Semantic Enhancement for Edge-Cloud Systems
- Post training 4-bit quantization of convolutional networks for rapid-deployment
- Feature Compression for Cloud-Edge Multimodal 3D Object Detection