Relational knowledge distillation trains a small "student" network to copy how a large "teacher" arranges examples relative to one another: which pairs of images the teacher places close together and which it places far apart. It does not copy the teacher's raw coordinates. This works well for image retrieval. On Cars 196, a ResNet-18 student trained this way reaches 82.50% Recall@1 and beats its own ResNet-50 teacher at 77.17% (Relational Knowledge Distillation). But the distances being copied are ordinary Euclidean distances computed in the teacher's high-dimensional embedding. The high-dimensional statistics literature shows these distances are systematically biased when noise differs from example to example. If example $i$ carries noise of total variance $r_i$, then every distance from $i$ is inflated by that amount, and a noisy example looks far from everything (Euclidean Distance Deflation Under High-Dimensional Heteroskedastic Noise). A student that faithfully copies these distances learns the teacher's noise level as if it were structure.
The proposal is to estimate each example's noise contribution in the frozen teacher's embedding space, subtract it from the pairwise distances, and distill the corrected distances. Because the teacher never changes, the estimate can be computed once, offline. Training then costs no more than standard relational distillation. The work tests whether this correction helps real students, when it helps, and whether much simpler fixes already capture the same benefit.
Core Claim
The observed squared distance between noisy teacher embeddings $y_i, y_j$ decomposes as
$$\|y_i - y_j\|^2 \approx \|x_i - x_j\|^2 + r_i + r_j ,$$
where $x_i$ is the clean signal component and $r_i$ is the per-example noise energy. The approximation holds because cross terms between independent noise and signal concentrate near zero in high dimension (Euclidean Distance Deflation Under High-Dimensional Heteroskedastic Noise).
Relational distillation's distance loss divides every distance by the batch mean. That removes a global scale, but it cannot remove the per-example terms $r_i + r_j$. The angle loss is also affected. The inner product $(y_i - y_j)\cdot(y_k - y_j)$ picks up $r_j$ at the vertex, which pushes all triangles toward equilateral as noise grows.
We claim three things. First, penultimate-layer embeddings of standard vision teachers are measurably heteroskedastic in this sense. Second, relational targets computed from them are biased in the predicted way. Third, distilling the deflated distances $\tilde d_{ij} = d_{ij} - \hat r_i - \hat r_j$ improves retrieval, and the gains should grow with the spread of $\hat r$ across examples, not with dimension alone.
The first claim is falsifiable. If the estimated $\hat r_i$ are nearly constant, the batch normalization already absorbs the bias, and the idea fails on real data.
This differs from its neighbors in specific ways. Knowledge Diffusion for Distillation also treats noise as the obstacle. It assumes the student's features are a noisy version of the teacher's and removes that noise with a trained multi-step generative denoiser. We instead assume the teacher's relational target is itself corrupted, and we correct it in closed form using a known geometric bias. Neither Relational Knowledge Distillation nor Simple Unsupervised Knowledge Distillation With Space Similarity (CoSS, which matches both rows and feature columns of the embedding matrix) contains any mechanism for per-example noise.
Research Program
The estimator is borrowed from the distance-deflation work, and cost is its main practical limitation. That work finds true neighbors by exploiting an invariance of the linear assignment problem: its optimal matching is unchanged when a constant is added to all costs in a row or column, which is exactly the $r_i$-shaped inflation. It then solves for $r_i$ from triangles of near neighbors, using $r_i \approx (d_{ij} + d_{ik} - d_{jk})/2$ when the true distances among neighbors $i, j, k$ are small. The assignment step costs $O(n^3)$ (Euclidean Distance Deflation Under High-Dimensional Heteroskedastic Noise).
We avoid paying this during training. The teacher is frozen, so we run the estimator offline on chunks of a few thousand training embeddings and take a median over triangles and chunks for each $\hat r_i$. We also build a cheaper variant that finds neighbors by off-diagonal inner products instead of by the assignment problem, and we compare it against the exact estimator on subsets.
The distillation loss is otherwise identical to relational distillation's distance and angle losses, with deflated teacher distances (and angles recomputed from them) as targets. An ablation also deflates the student side, although the student's low-dimensional embedding should carry less accumulated noise.
The analysis runs in two regimes. In the controlled regime we append synthetic nuisance dimensions with per-example variances drawn from a spread distribution to a clean teacher embedding. This makes $r_i$ known, so estimator accuracy and student degradation can be plotted against noise dimension and noise spread separately. In the natural regime we use unmodified 2048-dimensional ResNet-50 and ResNet-101 pooled features.
We also connect the findings to theory. High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws proves, in linear regression, that an optimal teacher should prune weak high-dimensional components so the student does not fit noise. Our correction acts on a different axis: per-example noise rather than per-direction noise. We will report whether the two interact, by combining deflation with pruning of low-signal principal directions.
Baselines
The naive reference is the same compact student trained on ground-truth labels alone with a triplet loss. Learning Metrics From Teachers: Compact Networks for Image Embedding reports that this gap is large: MobileNet-0.25 on CUB-200-2011 goes from 27.5% to 44.6% Recall@1 with distillation. We rerun this baseline with identical schedules.
The strongest existing competitor is relational distillation itself, with the distance and angle losses of Relational Knowledge Distillation. Its reported reference is 82.50% Recall@1 on Cars 196 for a ResNet-18 student from a ResNet-50 teacher. It is the right competitor because our method changes only its targets. We will reproduce it under our pipeline and compare against our reproduction, not the paper's number, keeping batch size, embedding dimension, loss weights, epochs and tuning budget identical, with at least three seeds.
We also include the relative-distance teacher loss from Learning Metrics From Teachers: Compact Networks for Image Embedding, which reports 76.6% Recall@1 on Cars-196 with a ResNet-101 teacher, and CoSS from Simple Unsupervised Knowledge Distillation With Space Similarity. For DiffKD we found no reported result on Lacuna for metric-learning retrieval. We will run its feature-denoising module on top of the same student and report its extra training time.
The ablation that isolates our key ingredient is masked-Gram matching. It distills only the off-diagonal inner products $\langle y_i, y_j \rangle$, which are unaffected by $r_i$ in expectation because the noise energy sits only on the diagonal. This baseline keeps the "remove the self-term" idea but throws away all norm information. Deflation must beat it to justify estimating $\hat r_i$ at all.
Beating relational distillation means a Recall@1 gain larger than the across-seed spread on at least two of CUB-200-2011, Cars 196 and Stanford Online Products. The gain must also be accompanied by the predicted signature: larger gains on examples with high $\hat r_i$.
Why not just…?
Why not just project teacher embeddings with PCA before distilling? PCA ranks directions by variance. Deep Unsupervised Feature Selection by Discarding Nuisance and Correlated Features shows that nuisance directions can carry more variance than signal. Dimensionality Reduction for Spectral Clustering reports PCA followed by spectral clustering at 0.75 normalized mutual information versus 0.87 for a jointly learned projection. More basically, PCA removes directions, while our bias is per example: noise spread isotropically but unevenly across examples survives any projection in proportion to its retained dimension. We still run PCA-then-relational-distillation at several ranks. If it matches deflation in the natural regime, that tells us real teacher noise is directional rather than per-example, which would redirect the work toward pruning.
Why not just normalize embeddings to unit length? Unit normalization changes $r_i$ but does not equalize it. After normalization, a noisy example's signal component shrinks, which distorts its angles instead. CoSS also argues that normalization collapses the teacher's geometry and reports gains from undoing it (62.35% vs 61.40% ImageNet top-1 against a normalized method, Simple Unsupervised Knowledge Distillation With Space Similarity). We include cosine-distance relational distillation as a baseline and expect it to show the same high-$\hat r$ error pattern.
Why not just match off-diagonal inner products and skip estimation? This is the real threat, because it removes the $r_i$ bias for free. Its cost is that it discards norms, which in retrieval embeddings can encode confidence and typicality. The masked-Gram ablation settles the question directly. If it matches deflation, we report that the simple fix suffices, and the contribution narrows to the diagnosis and the controlled analysis.
Why not just use DiffKD's learned denoiser? It denoises student features, not teacher targets, and it needs at least five reverse diffusion steps (Knowledge Diffusion for Distillation). The two corrections are complementary. We therefore test combining them, as well as comparing them at equal wall-clock time.
Proposed Experiments
The first decisive check is diagnostic and precedes any training. We estimate $\hat r_i$ for ResNet-50 and ResNet-101 penultimate features on CUB-200-2011, Cars 196 and Stanford Online Products. We report its spread relative to typical nearest-neighbor distance, and whether high-$\hat r$ examples are the ones retrieved poorly by the teacher. A near-constant $\hat r$ ends the natural-regime claim.
The second check uses the synthetic nuisance regime. The prediction is that relational distillation's Recall@1 falls as noise spread increases at fixed total noise, while the deflated version stays flat. Degradation under increasing dimension with constant per-example variance should be largely absorbed by the batch normalization already present.
The third check runs the full retrieval comparison at teacher dimensions 512 and 2048 with 128-dimensional students, where the Cars 196 cross-dimensional setting reported by Relational Knowledge Distillation is the anchor. Alongside Recall@1 overall, we report Recall@1 split by $\hat r$ quantile and training time.
A fourth, lower-priority check probes the known weakness that strong relational constraints hurt out-of-distribution transfer. We distill on Cars 196 and evaluate on CUB-200-2011, testing whether copying the teacher's noise explains part of that over-specialization.
References
- Relational Knowledge Distillation
- Euclidean Distance Deflation Under High-Dimensional Heteroskedastic Noise
- Knowledge Diffusion for Distillation
- Simple Unsupervised Knowledge Distillation With Space Similarity
- High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws
- Learning Metrics From Teachers: Compact Networks for Image Embedding
- Deep unsupervised feature selection by discarding nuisance and correlated features
- Dimensionality Reduction for Spectral Clustering