When a small model is trained to copy a larger one, a technique called knowledge distillation, there is a choice of what to copy. The student can copy the teacher's final predictions, its internal activation vectors, or its attention maps. An attention map is the table saying how strongly each token looks at every other token. Separately, low-rank adaptation (LoRA) fine-tunes a frozen network by adding a small trainable correction of the form $BA$ to chosen weight matrices. KD-LoRA (KD-LoRA) combines the two: it trains LoRA adapters on a compact student using only the teacher's output predictions. The one attempt to supervise LoRA students internally, LoRAILD, matched hidden vectors and produced no significant gain (LoRAILD). This proposal argues that the internal target for a LoRA student should instead be attention maps. There is a structural reason for this beyond the empirical robustness of attention distillation. A low-rank change to the query and key weights produces a low-rank change in the matrix that scores token pairs. That means the attention gap between teacher and frozen student can be measured before training, head by head, for how much of it rank-$r$ adapters can close at all.
The claim has two parts, and both can be tested. First, for a compact student with a frozen backbone, distilling attention maps into adapters placed on the query and key projections beats logit-only KD-LoRA at matched trainable parameters and matched tuning effort. It should come close to full-parameter attention distillation. Second, a cheap pre-training diagnostic, which we call attention reachability, predicts two things: which heads and layers benefit from this, and where the adapter rank budget should go. If reachability fails to predict outcomes, the structural story is wrong, even if the aggregate accuracy gain holds.
Core Claim and Mechanism
For one attention head, the pre-softmax score between tokens $i$ and $j$ is $s_{ij} = x_i^\top W_q W_k^\top x_j/\sqrt{d_h}$. Here $x_i$ is the token's input vector to the layer, $W_q, W_k$ are the head's query and key projections, and $d_h$ is the head width. Adding LoRA corrections $\Delta W_q = B_qA_q$ and $\Delta W_k = B_kA_k$ of rank $r$ changes the bilinear form $M = W_qW_k^\top$ by
$$\Delta M = \Delta W_q (W_k + \Delta W_k)^\top + W_q \Delta W_k^\top, \qquad \operatorname{rank}(\Delta M) \le 2r.$$
The bound holds because each term inherits the rank of its LoRA factor. So the full set of attention corrections a rank-$r$ query/key adapter can express is exactly the set of rank-$2r$ corrections to $M$. This holds if the layer's inputs are held fixed. When earlier layers are also adapted, the inputs change, so the bound is only approximate across the whole network.
This gives a precise question to ask before any training. Collect student inputs $X$ and teacher score matrices $S_T$ on a few hundred task examples. We then fit the best rank-$2r$ correction $\Delta$, using row-centered scores because softmax ignores constant shifts within a row. From that fit we define the reachability of head $h$ as
$$\rho_h(r) = 1 - \frac{\min_{\operatorname{rank}(\Delta)\le 2r}\lVert S_T - X(M+\Delta)X^\top\rVert^2}{\lVert S_T - XMX^\top\rVert^2}.$$
This is the fraction of the teacher–student score gap that a rank-$r$ adapter could remove in principle. There is no analogous closed-form object for hidden-state matching. There, the adapter must also absorb scale differences, which LoRAILD had to handle with added projection machinery.
The placement point matters in practice. An attention-map loss sends no gradient to a value-projection adapter in the same layer. KD-LoRA places adapters on the query and value projections, so naively adding an attention loss to it would train only half of its adapters on the new signal. The design here puts attention-supervised adapters on query and key, and leaves value (and optionally feed-forward) adapters to the logit and task losses.
Relation to Prior Work
Three papers motivate the proposal.
- How to Distill your BERT compared distillation objectives head to head. It found mean-squared error on attention maps (Att-MSE) the most robust choice across student initializations, while hidden-state transfer was sensitive to initialization. All its students were fully trained, not adapter-tuned. We adopt its objective and ask whether its robustness survives when only 1% of parameters move.
- Compressing Large-Scale Transformer-Based Models surveys BERT compression. It reports that attention transfer preserves linguistic structure better than logits alone.
- KD-LoRA supplies the setting: a DistilBERT student with LoRA and a fine-tuned BERT teacher, retaining about 97% of full fine-tuning performance with logits only. It leaves internal supervision untried.
Two papers constrain the design.
- LoRAILD used LoRA to align intermediate hidden vectors and found no significant improvement over vanilla KD. That negative result is why we reject hidden-state targets. It is also why we insist on multi-seed significance testing, since its failure was a failure to separate from noise.
- LoLCATs turns softmax attention into linear attention by matching attention-layer outputs, then tunes with LoRA in a separate step. Its authors observed that matching outputs worked better than matching attention weights. This apparently contradicts our approach, but their student was a linear-attention layer that cannot reproduce spiky softmax distributions. Our student is a softmax transformer of the same family as the teacher, where the maps are directly comparable. We still test output matching as an ablation rather than assume the contradiction away.
Two further works bound the scope.
- LoRaDA adds a low-rank matrix directly to the attention map so attention can go negative. It is trained only on the task, without a teacher. It is a natural second adapter type to supervise with teacher maps, but we treat it as an extension, not the core.
- Recover-LoRA repairs quantized or pruned small language models with LoRA and logit distillation, recovering 5–17% accuracy. It provides our second testbed. Teacher and student there share an architecture, so heads align one-to-one and the reachability diagnostic is cleanest.
Research Program
We first compute reachability curves $\rho_h(r)$ for every student head, for $r \in \{1,2,4,8,16\}$. The fit uses alternating least squares on centered scores from a few hundred examples, with layers treated block by block on frozen inputs. This characterizes where the teacher–student attention gap lives and how low-rank it is. It is a cheap analysis that stands on its own.
We then train students with a combined loss: task cross-entropy, teacher logit KL, and Att-MSE on attention probabilities. The Att-MSE term is applied only to query/key adapters in mapped layers. For the six-layer DistilBERT student, the default is to map student layer $l$ to teacher layer $2l$. A last-layer-only mapping is an ablation.
Third, we use reachability to allocate a fixed adapter budget: more rank to heads with a large gap and high reachability, none to heads whose gap is either tiny or unreachable at any small rank. This is compared against uniform rank at identical parameter count.
Finally, we test the predictive claim directly. After training, we check whether per-head reachability correlates with the per-head reduction in Att-MSE that was actually achieved. We also check whether layers with high aggregate reachability are the ones whose supervision removal hurts accuracy most.
The quadratic cost of attention maps is modest at GLUE lengths. For the language-model testbed we compute the loss on a random subset of query rows per head, which gives an unbiased estimate of the full-map loss.
Baselines
Naive reference. The DistilBERT student with vanilla LoRA and task loss only, no teacher. It sets the floor that any distillation must clear.
Strongest parameter-efficient competitor. KD-LoRA itself: LoRA on query and value, task plus logit KL, with the fine-tuned BERT teacher. It is the published method for exactly this problem and is the direct thing we claim to beat. We re-run it in our codebase at matched trainable parameters. Following A Unified Study of LoRA Variants, which found LoRA results highly sensitive to learning rate, every method gets the same learning-rate and temperature sweep, the same steps, and five seeds. The file gives KD-LoRA only in relative terms (about 97% of full fine-tuning and 98% of standard LoRA on GLUE); no reported absolute GLUE average for KD-LoRA was found on Lacuna. Beating it means a GLUE average gain whose confidence interval across seeds excludes zero, not a single-seed margin.
Full-parameter ceiling. Full-parameter Att-MSE distillation into a six-layer student, as in How to Distill your BERT. That paper reports an average GLUE score of 81.3 for its optimized six-layer student, against 78.5 for DistilBERT and 81.0 for MiniLM. Closing most of the gap from KD-LoRA toward this regime, while training about 1% of parameters, would be the headline result.
Internal-target competitor. LoRAILD-style hidden-state matching through LoRA adapters, re-implemented at the same budget. It tests whether the target (maps versus vectors) is what matters.
Ablations isolating the key ingredient. - Attention loss routed to query/value adapters, as KD-LoRA's placement would imply, versus query/key. - Attention-output matching in place of map matching, to address LoLCATs. - Uniform versus reachability-allocated rank at equal parameter count. - Randomly allocated rank, which controls for the mere fact of non-uniformity.
Why not just…?
Why not just keep KD-LoRA's logit-only distillation and tune it well? This is the most serious threat. KD-LoRA already retains most of full fine-tuning, and a thorough sweep might close the remaining gap. The evidence against it is indirect. Wang et al. found logit KD brittle: QNLI accuracy under vanilla KD moved from 66.5 to 86.2 depending only on which teacher layers initialized the student, while Att-MSE stayed consistently strong. Our student initialization is fixed, though, so that brittleness may not bite. The matched-sweep, five-seed comparison settles it. If logit-only KD-LoRA lands within noise of our method, we will report that attention supervision adds nothing to adapter students and keep the reachability analysis as the contribution.
Why not just distill attention into the full student and skip adapters? Full-parameter attention distillation is the known-good recipe, and it is our ceiling baseline, not a rival. The question is whether a frozen student plus rank-8 query/key adapters gets close to it. KD-LoRA reports roughly 75% lower training memory than full fine-tuning and half the trainable parameters of teacher-side LoRA, and those savings matter only if accuracy holds. If the adapter version falls far short, the reachability curves should show why: low $\rho_h(r)$ at small $r$. That is itself a finding about when parameter-efficient distillation can work.
Why not match hidden states, which is cheaper than quadratic attention maps? LoRAILD did exactly this and found no significant gain over vanilla KD. Wang et al. found hidden-state transfer sensitive to initialization. We re-run it at matched budget so the comparison is direct rather than borrowed. For sequence lengths in GLUE, the attention-map memory cost is small. For long sequences, row subsampling bounds the cost.
Why not just raise the rank of the student's LoRA until task loss alone suffices? Rank is cheap to increase, and the unified LoRA study suggests well-tuned vanilla LoRA is hard to beat. Our uniform-rank sweep with and without the attention loss answers this directly. The reachability view predicts that the attention loss helps most at low rank. If the gain survives at rank 16 and above, that is evidence the teacher supplies information the task alone does not.
Proposed Experiments
Testbed 1: GLUE with encoders. We use the GLUE benchmark (GLUE) with a fine-tuned BERT-base teacher and a DistilBERT student. Both share a tokenizer and 12 heads per layer, so maps align by token and head. We report per-task metrics (Matthews correlation for CoLA, Pearson/Spearman for STS-B, accuracy or F1 elsewhere) and the GLUE average, with seed means, standard deviations, and paired significance tests. Trainable parameters, peak training memory, and student inference time are reported alongside, to confirm the efficiency profile KD-LoRA established is preserved.
Testbed 2: repairing a degraded language model. Following Recover-LoRA, we quantize or prune a small decoder model and repair it with LoRA against its full-precision self. We compare logit-only repair with attention-map repair, evaluating on the commonsense suite (PiQA, ARC, HellaSwag, OpenBookQA, WinoGrande, BoolQ) and MMLU (MMLU). The file reports Recover-LoRA's 5–17% recovery only as a range; we will run its logit-only recipe ourselves at the same synthetic-data budget. Because heads align exactly here, this is where the reachability diagnostic faces its cleanest test.
Decisive checks. The proposal succeeds on its main claim if attention-supervised query/key adapters beat matched, tuned KD-LoRA on the GLUE average with seed-level significance, and beat the hidden-state variant. It succeeds on its structural claim if per-head reachability, computed before training, correlates positively and substantially with per-head Att-MSE reduction after training. It must also show that reachability-guided rank allocation beats both uniform and random allocation at equal parameter count.
It fails informatively in two ways. If gains appear with query/value placement as well as query/key, the rank-bound explanation is not what drives them. If attention-output matching equals map matching, the LoLCATs observation generalizes to same-family students, and the case for maps specifically weakens.
A small extension, reported only if the core holds, supervises LoRaDA's additive attention matrix with the residual between teacher and student maps. LoRaDA reports 67.2% versus 39.5% for standard LoRA on commonsense reasoning at a 0.10% budget with LLaMA-7B. The extension asks whether a teacher can tell that module which token pairs to push negative.
References
- KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation
- Aligning Sizes of Intermediate Layers by LoRA Adapter for Knowledge Distillation
- How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives
- Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
- LoLCATs: On Low-Rank Linearizing of Large Language Models
- LoRaDA: Low-Rank Direct Attention Adaptation for Efficient LLM Fine-tuning
- Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation
- A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Measuring Massive Multitask Language Understanding