When a frozen pretrained model is adapted with a small add-on module, such as a low-rank update or an adapter, it can only recombine features the backbone already computes. If the target domain needs features the backbone never learned, such as histology textures or long-tail facts, the add-on hits a ceiling. This failure is called representation rigidity. There is one clean hint about what breaks the ceiling: MEFT found that knowledge-heavy tasks needed adapters of about 10% of backbone size. Its 10% sparse adapter on LLaMA-7B reached 0.413 exact match on Natural Questions, against 0.305 for LoRA. MEFT got there by offloading the adapter to CPU memory, so it was limited by the bus between CPU and GPU. Nobody knows what happens at 30%, 100%, or several times backbone size, because no system can train adapters that large efficiently.
We propose to build one and use it to answer that question. The adapter is a very large key–value memory. Each token scores itself against millions of learned keys, keeps its top few matches, and adds a weighted sum of the matching values to its hidden state. The keys and values are split across many GPUs, and each shard is updated only on the GPU that owns it. Adapter parameters therefore never cross the network. Only token activations and a few per-token numbers do, so communication does not grow with adapter size. Two things make this work: the communication-avoiding sparse matrix layouts from distributed linear algebra, and a fused kernel for the row-wise top-k and normalization step that existing distributed sparse kernels cannot handle. With the system in hand, we trace how accuracy scales with adapter capacity on domains at different distances from pretraining. That tells us whether rigidity is a capacity problem that more memory fixes, or something a frozen backbone cannot escape.
Core Claim
A sparse memory adapter computes, for a token's hidden state $h \in \mathbb{R}^{d}$,
$$y = h + \mathrm{softmax}_{k}\!\left(h K^{\top}\right) V,$$
where $K, V \in \mathbb{R}^{m \times d}$ are $m$ learned keys and values, and $\mathrm{softmax}_k$ keeps the $k$ largest scores in the row, normalizes them, and zeroes the rest. This one operator covers several existing methods. With keys initialized from training features and values from one-hot labels, it is Tip-Adapter's cache. With learned keys and sparse activation, it is essentially MEFT's adapter.
For a batch of $n$ tokens, the computation has three steps. It first computes scores (dense matrix times dense matrix), then selects the top $k$ per row (a sparse mask), then multiplies the sparse mask by $V$ (sparse matrix times dense matrix). The first and last steps are the sampled dense–dense multiplication (SDDMM) and sparse-times-dense multiplication (SpMM) pair that Distributed-Memory Sparse Kernels for Machine Learning fuses into its FusedMM operation. That paper reuses replicated dense operands across both steps in 1.5D and 2.5D layouts. It explicitly cannot place a row-wise nonlinearity such as softmax between the two steps.
Our claim is that the nonlinearity needs only a small reduction of per-row statistics along one grid dimension. That is $O(nk)$ candidate scores for top-k, or $2n$ scalars for softmax in the online max-and-sum style. Given this reduction, the adapter's per-step communication is $O(nd)$ plus that small term, independent of $m$.
The competing approach is to shard a dense adapter as ZeRO-style data parallelism does. It gathers every weight forward and backward and reduce-scatters gradients, which costs on the order of $md$ per step. That count comes from moving each parameter a constant number of times. Our layout therefore wins exactly when memory size exceeds batch size ($m \gg n$). That is the regime where the rigidity question is still open.
Research Program
System. We adopt the geometric view of matrix multiplication as a three-dimensional cube indexed here by token, memory slot, and hidden dimension. Exploiting Multiple Levels of Parallelism in SpGEMM describes this cube. Our processor grid places memory slots along one axis, token groups along a second, and uses a replication factor $c$ on the third. This follows the memory-for-bandwidth trade that 2.5D algorithms make (Communication-Avoiding Symmetric Eigensolver).
With $c=1$, tokens are replicated to every slot shard. Larger $c$ splits the token batch into groups and replicates slot shards, which trades memory for less activation broadcast. The fused kernel follows the owner-computes rule. Each GPU computes its local scores and a local top-k. The GPUs then run a small all-reduce along the slot axis to find the global per-row threshold and normalizer. Each GPU finishes the SpMM on its own values and contributes to a reduction of $y$. The backward pass mirrors this, and the gradients for $K$ and $V$ stay on their owners.
Two known problems from sparse linear algebra carry over directly. The first is load imbalance: a few memory slots will attract most tokens, just as power-law graphs overload some processors (Communication-Avoiding and Memory-Constrained SpGEMM at Extreme Scale). We handle it by replicating hot slots adaptively, using the per-slot hit counts that the kernel already computes. The second is memory blow-up in large caches. For those we reuse that paper's batching and hash-based local accumulation. Scoring all $m$ keys is still $O(nmd)$ arithmetic, split over $p$ devices. We keep this exact version as the reference and test a routed variant that scores only candidate key clusters, following MEFT's key-expert grouping.
Science. Once the system runs, we sweep adapter memory size from LoRA-scale up to several times backbone parameters. Each sweep is on a frozen backbone, and we cover several domains that sit at different distances from pretraining. For each domain we fit accuracy against capacity. There are three possible outcomes. Accuracy may keep climbing, in which case rigidity is a capacity problem. It may plateau below full fine-tuning, in which case the frozen features themselves are the limit. Or it may overfit, in which case capacity must be paired with regularization. The high-rank reparameterization of ABBA supplies the parameter-matched dense comparison. It tells us whether sparse memory capacity differs from simply having more effective rank.
Baselines
The naive reference point is the frozen backbone with a linear probe. For vision–language backbones we add text-anchored linear probing. Few-shot Adaptation of Medical Vision-Language Models reports 55.6% one-shot accuracy on NCT-CRC for that probe, versus 46.05% for CoOp. If memory adapters cannot beat this probe on histology, the premise fails for that domain.
On knowledge acquisition, the strongest existing method is MEFT with its 10% adapter, which reports 0.413 exact match on Natural Questions (LoRA 0.305, Parallel Adapter 0.387). We rerun MEFT and our method on the same LLaMA-7B backbone, the same data, and the same optimizer-step budget. The 10% point must match MEFT's reported quality. That check confirms correctness. The real test is whether larger memory sizes, which only our system reaches in practical wall-clock time, exceed 0.413.
On few-shot vision–language transfer, the reference is Tip-Adapter-F at 65.51% on 16-shot ImageNet. On VTAB-1k it is Res-Tuning at 74.10% average accuracy, and MoSA, which reports gains of 2.43% over full fine-tuning and 1.06% over standard adapters. Beating these requires a gain concentrated in the Specialized and Structured groups, where rigidity should bite, and not merely in the Natural group.
The strongest systems baseline is a dense adapter of identical parameter count, sharded with ZeRO-3-style fully sharded data parallelism. No reported result for this configuration was found on Lacuna, so we run it ourselves. It gets the same GPUs and the same per-GPU memory, and we tune its bucket sizes and prefetching for fairness. We also compare against a mixture-of-experts style layout with expert parallelism, which clusters slots into experts and sends tokens by all-to-all. The FusedMM kernel without our normalization step serves as the lower bound on communication. Here "beating" means lower time per step at equal accuracy, with the advantage growing as $m$ grows, as the scaling argument predicts.
Three ablations isolate the key ingredients. The first replaces the fused top-k reduction with gathering full score rows, which isolates the normalization step. The second fixes $c=1$, which isolates the 2.5D replication. The third disables hot-slot replication, which isolates the load-balancing contribution.
Why not just…?
Why not just shard a dense adapter with ZeRO/FSDP? At the 10% scale MEFT studied, this may well be adequate, and it is a real threat to the systems half of the proposal. Its per-step communication grows with adapter size, while ours grows with batch activations. The capacity sweep will locate the crossover. If the crossover lies beyond any size that improves accuracy, we will report that plainly, and the contribution becomes the scaling study alone.
Why not just use mixture-of-experts expert parallelism? Coarse expert routing moves about as many bytes as our layout. It gives up exact top-k over all slots and suffers the same hot-expert imbalance. We run it as a baseline under matched capacity. If it matches both accuracy and speed, our kernel adds little. The exact-versus-routed comparison will show whether fine-grained selection matters for breaking rigidity.
Why not just unfreeze the backbone and fine-tune it with Megatron-style tensor parallelism? Path-CLIP reports catastrophic forgetting and overfitting from full fine-tuning in low-data pathology. ABBA reports 84.08% for its adapter versus 82.39% for full fine-tuning on commonsense reasoning with Llama-3.2 3B. We still include full fine-tuning as the upper reference on every domain. Our capacity curves are read against it: the question is whether frozen-backbone capacity closes the gap to full fine-tuning.
Why not just raise LoRA rank or use ABBA? Those methods increase effective rank, not the number of stored, retrievable directions. That is why ABBA is our parameter-matched control and not a straw man. If ABBA matches memory adapters at equal parameters on Natural Questions and VTAB-Structured, then sparsity is not the ingredient that matters.
Proposed Experiments
The kernel is validated first on the sparse matrices used by FusedMM (amazon-large, uk-2002, eukarya, arabic-2005, twitter7). We run graph-attention-style SDDMM–softmax–SpMM on them and report communication time against unfused FusedMM, which itself reported a 30% communication reduction for 1.5D on 256 Cori nodes. The adapter system is then benchmarked by strong and weak scaling in memory size $m$ against the sharded-dense and expert-parallel baselines.
The scientific sweep runs on four settings. Natural Questions with LLaMA-7B tests knowledge. VTAB-1k with a ViT tests structured and specialized vision. PCam and NCT-CRC with CLIP test a far domain. 16-shot ImageNet with CLIP is the near-domain control.
The decisive check is the shape of the accuracy-versus-capacity curve on far domains relative to near ones. A continued rise on PCam and VTAB-Structured beyond 10% capacity, with a plateau on ImageNet, would show that rigidity is largely a capacity problem that distributed sparse memory can remove. A flat curve past MEFT's 10% point would show the opposite, that the frozen features themselves are the limit. Either result is informative, and the measurement is only practical with parameters that stay on the GPU that owns them.
References
- MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter
- Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
- Distributed-Memory Sparse Kernels for Machine Learning
- Exploiting Multiple Levels of Parallelism in Sparse Matrix-Matrix Multiplication
- A communication-avoiding parallel algorithm for the symmetric eigenvalue problem
- Communication-Avoiding and Memory-Constrained Sparse Matrix-Matrix Multiplication at Extreme Scale
- ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
- Few-Shot Adaptation of Medical Vision-Language Models
- Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone
- Split & Merge: Unlocking the Potential of Visual Adapters via Sparse Training
- Bridging the Pathology Domain Gap: Efficiently Adapting CLIP for Pathology Image Analysis with Limited Labeled Data