Multi-Scale Consensus Training for Compact Siamese Embeddings

A Siamese network is a pair of identical, weight-sharing encoders that map two inputs to vectors and judge them similar when the vectors are close. Such networks underpin verification and retrieval: face and speaker checks, product search, tracking. Practitioners can make them better in three ways, and each has a cost. They can use a wider embedding, but past some width extra dimensions start fitting noise. They can fuse features from several depths of the network, but concatenating those features inflates the width further. They can average several independently trained networks, but inference then costs several times as much. This note asks whether the benefit of multi-scale fusion and ensembling in metric learning is really an inference-time benefit or a training-time one. If it is a training-time benefit, a single small embedding can absorb it.

The proposal is to attach small embedding heads at several depths of one shared Siamese backbone. The heads are trained to agree on the pattern of pairwise similarities within each batch, not on class labels. At test time every head except one fixed-width head is thrown away. The claim is that this retained head, at a modest width such as 128 dimensions, matches or beats three alternatives at no extra inference cost: a single head of any width, a multi-scale concatenation projected to the same width, and much of the accuracy of a multi-model ensemble. It should also move the point at which widening the embedding stops helping.

Core Claim and Relation to Prior Work

The closest existing work is Collaborative Learning for Deep Neural Networks. It trains several classifier heads off a shared backbone, makes them agree through a consensus soft-label loss, rescales the gradients where the heads branch, and discards the extra heads at test time. It reports 22.29% ImageNet top-1 error for ResNet-50 against a 23.47% baseline, with 1.5x rather than 4x training memory. This paper supplies our method, but it is built entirely around class probabilities. Metric learning has no fixed label set to agree on, and the useful signal is the geometry of distances between items.

We replace label consensus with relational consensus: agreement on who-is-near-whom within a batch. We also place heads at different depths on purpose, so that their diversity comes from scale rather than only from random initialization.

Two further papers constrain rather than supply the design. Multiple Object Tracking via Feature Pyramid Siamese Networks shows that pooling pyramid features at several levels, with a contrastive loss per level, beats a last-layer Siamese baseline on MOT17 (it reports MOTA 44.9, IDF1 48.4). However, it keeps the concatenated vector at inference and never asks whether that larger vector saturates. Teacher-Students Knowledge Distillation for Siamese Trackers compresses Siamese trackers from 361.8 MB to 19.7 MB. Its pipeline is multi-stage: a pretrained teacher followed by a reinforcement-learning search over layer compression ratios. We aim for a single training stage with no external teacher.

The motivation for caring about width comes from the saturation literature. MTGAN found 512 dimensions optimal for triplet speaker embeddings and saw a slight degradation at 1024. The Role of Context Types and Dimensionality in Learning Word Embeddings found that larger dimensions raise intrinsic scores while hurting downstream tasks.

The direct evidence for saturation in visual metric learning is thin in this literature. The MTGAN drop is described as "slight," and Deep Metric Learning via Lifted Structured Feature Embedding sweeps 64 to 512 dimensions, but the file does not give the curve. Measuring the saturation curve carefully is therefore part of the program, not an assumption.

Research Program

The model has a shared backbone, for example an ImageNet-pretrained ResNet-50. Global-average-pooled features from stages 2, 3, and 4 each feed a small projection to a $D$-dimensional, unit-normalized embedding. A fourth head projects the concatenation of all pooled stages to $D$ dimensions. Every head is trained with the same pairwise metric loss (a triplet loss with standard mining). On top of that sits a consensus term.

For head $h$ and a batch of $B$ items, let $s^h_{ij}$ be the cosine similarity between items $i$ and $j$. Let $P^h_{i\cdot}$ be the softmax over $j \neq i$ of $s^h_{ij}/\tau$, where $\tau$ is a temperature. Let $\bar P_{i\cdot}$ be the average of these distributions over the $H$ heads, with gradients stopped. The consensus loss is

$$\mathcal{L}_{\text{cons}} = \frac{1}{B}\sum_{i=1}^{B}\sum_{h=1}^{H} \mathrm{KL}\!\left(\bar P_{i\cdot}\,\|\,P^h_{i\cdot}\right),$$

and the total objective is $\sum_h \mathcal{L}^h_{\text{metric}} + \lambda\,\mathcal{L}_{\text{cons}}$, with $\lambda$ a weight. Following Collaborative Learning, gradients entering the shared backbone from the $H$ heads are scaled by $1/H$. This avoids the gradient conflict that paper identified.

Because the loss compares neighbor distributions rather than coordinates, heads at different depths can live in different coordinate systems while still agreeing on neighborhood structure. This is the property a Siamese distance needs.

At test time we keep one head. The default is the fused projection head, but which head is kept is an ablation. The retained model has the same width as a plain $D$-dimensional Siamese network and nearly the same encoder cost.

To study saturation, we sweep $D \in \{64, 128, 256, 512, 1024\}$ for every method. Alongside Recall@1 we report the effective dimensionality of the test embeddings:

$$d_{\text{eff}} = \frac{\left(\sum_k \lambda_k\right)^2}{\sum_k \lambda_k^2},$$

where the $\lambda_k$ are the eigenvalues of the embedding covariance. This number is the dimension the data actually occupies. We use it for a diagnostic hypothesis: saturation appears when nominal width grows while $d_{\text{eff}}$ stalls and the extra directions carry train-set-specific variance.

We test two predictions. First, consensus training should raise the $D$ at which Recall@1 peaks and flatten the decline after it. Second, it should keep $d_{\text{eff}}$ growing with $D$ for longer than the baselines do.

Baselines

The naive reference is a standard single-head Siamese network trained with the same triplet loss, backbone, and schedule, reported across the full width sweep. Its best width across the sweep, not just its width at 128, is the bar the retained head must clear. This rules out the reading that we merely picked a bad width for the baseline.

The strongest published single-model competitor is Multi-level Distance Regularization. Multi-level Distance Regularization for Deep Metric Learning regularizes normalized pairwise distances toward a few learned similarity levels alongside a triplet loss. It reports Recall@1 of 68.8% on CUB-200-2011 (against a 57.3% triplet baseline) and 88.5% on Cars-196 (against 76.2%). On Stanford Online Products, the reported figure we have is roughly 62% Recall@1 for lifted structured embedding with a GoogLeNet backbone (Lifted Structured Feature Embedding). For In-Shop Clothes we found no reported result on Lacuna.

We will re-run MDR in our exact setting: same backbone, image resolution, epochs, batch size, and a matched hyperparameter search budget of the same number of trials per method. Beating it means a higher Recall@1 at equal $D$ on at least three of the four visual benchmarks, averaged over three seeds with non-overlapping intervals. Because MDR regularizes the output distances while our method regularizes through multi-depth agreement, we will also run MDR combined with consensus to check whether the two are complementary.

The multi-scale fusion baseline follows the recipe of the feature pyramid Siamese network. We use per-level metric losses and a concatenated descriptor, run both at full concatenated width and with a learned projection to $D$, and train both without consensus. The ensemble upper reference is $M = 4$ independently seeded single-head Siamese networks with averaged cosine similarities, reported with its $M$-fold inference cost.

The key ablations isolate the ingredient we care about. The first keeps all heads and all per-head metric losses but sets $\lambda = 0$. This is plain deep supervision, and it tells us whether agreement matters or extra losses alone do the work. The second places all heads at the final stage, so that only seed diversity remains, which tests whether scale diversity matters. The third replaces the relational KL with coordinate-wise mean-squared agreement.

Why not just…?

Why not just ensemble several independent Siamese networks and average their distances? This is the accuracy ceiling we measure against, not a competitor at equal cost. Four networks mean four forward passes and four stored vectors per item. Collaborative Learning reports that the ensemble needs 4x memory against 1.5x for shared-backbone training. Real-time settings like the 265 FPS tracker in the Siamese distillation work cannot absorb that. The claim that matters is the fraction of the ensemble's gain over the naive baseline that the retained single head recovers. We call the result useful if it recovers at least half.

Why not just widen the final embedding? It is possible that visual metric learning simply does not saturate by 1024 dimensions. The evidence cited above comes from speech and word embeddings and shows only a modest effect. The width sweep settles this directly. If the naive baseline keeps improving up to 1024 dimensions, the dimensionality framing of this proposal weakens. The remaining claim is then an efficiency one: equal accuracy at a smaller $D$ means smaller retrieval indexes and faster search, the axis that Angular Dispersion Accelerates k-Nearest Neighbors Machine Translation shows matters for approximate nearest-neighbor search.

Why not just concatenate multi-scale features and project them to $D$ dimensions? This is the most serious threat. Pooling intermediate features adds almost no inference cost, so the "zero overhead" argument does not apply to it. That is why projected fusion without consensus is a primary baseline, and why the fused head is also a candidate retained head in our own method. If projected fusion matches consensus training, the honest conclusion is that multi-scale benefit is an inference-time effect and the consensus term is unnecessary. The $\lambda = 0$ ablation with the fused head retained is exactly the experiment that decides this.

Why not just distill an ensemble into a student afterwards? Two-stage distillation of the independent ensemble into a single $D$-dimensional student is a real alternative. We will run it using the same relational KL as the distillation loss, at matched total training compute (teacher plus student cost against our single run). If it wins at matched compute, our contribution shrinks to a cheaper training recipe. If it wins only with extra compute, the single-stage method is the practical choice.

Why not just add a dispersion regularizer? Consensus might help only by spreading embeddings apart. The sliced angular dispersion regularizer from the k-NN translation work is a direct test. We add it to the naive baseline and check whether it reproduces both the accuracy gain and the $d_{\text{eff}}$ gain.

Proposed Experiments and Decisive Checks

The visual benchmarks are CUB-200-2011, Cars-196, Stanford Online Products, and In-Shop Clothes, all under standard splits. The metrics are Recall@1, 2, 4, and 8 and $d_{\text{eff}}$, with Recall@k meaning the fraction of queries whose nearest $k$ neighbors include a same-class item. A second modality tests whether the effect is specific to images. Here we train speaker embeddings on LibriSpeech and evaluate zero-shot on TIMIT with equal error rate, the operating point where false accepts equal false rejects. MTGAN reports 1.81% equal error rate there at its best width of 512. Because our encoder will differ from MTGAN's, our comparison is the internal one: naive versus consensus heads across the same width sweep. We report per-query encoder FLOPs, embedding bytes, and approximate-nearest-neighbor search time on Stanford Online Products for every method.

Three outcomes would decide the question. The retained consensus head must beat projected fusion without consensus at equal $D$. If it does not, the idea fails. The same-depth-heads ablation must lose most of the gain, which would show that scale diversity, not just extra heads, is the ingredient. Finally, the Recall@1-versus-$D$ curve for consensus training must peak later or decline less than the naive curve, with $d_{\text{eff}}$ tracking that difference. Only this last result supports the claim about saturation. Without it, the work stands as a cheap route to ensemble-like metric embeddings and nothing more.