Zero-shot image classifiers such as CLIP recognize categories they were never trained on. They do this by comparing an image's embedding to text embeddings of prompts like "a photo of a dog". Their image encoders are built almost entirely from multiplications. A separate line of work, adder networks (AdderNets), replaces each multiply-and-sum with a negative Manhattan distance, $-\sum_i |x_i - w_i|$. That operation needs only subtraction, a sign flip, and addition, which is far cheaper in custom silicon. This note proposes to distill a frozen CLIP image encoder into an adder student and to run the whole on-device zero-shot path without multiplications. The text side is handled by precomputing class prompts once, offline. The central question is whether adder features, which the literature says organize space by distance to class centers rather than by angle, can keep the angular relationship to text that zero-shot transfer depends on.
The motivation is a specific worry that existing numbers cannot settle. On supervised ImageNet an adder ResNet-50 loses only 1.3 points (74.9% vs 76.2%). With kernel distillation it even gains 0.6 points (AdderNet; Kernel Based Progressive Distillation). Supervised accuracy only requires that training classes separate. Zero-shot accuracy on unseen concepts requires that the whole embedding geometry match a text space the student never optimized against. We claim the supervised gap can badly understate the zero-shot gap for adder students. We further claim that distilling the teacher's image-to-text similarity structure over a large bank of fixed text anchors closes most of this excess gap, while plain embedding regression does not.
Core Claim
Our first claim is measurable. When an adder and a multiplicative student of matched size are distilled from the same frozen CLIP teacher on the same unlabeled images, the adder student will show a larger drop on concepts absent from the distillation images than on concepts present in them. We call this difference the excess zero-shot gap. Our second claim is methodological. Replacing the regression target (match the teacher's embedding vector) with a relational target (match the teacher's softmax distribution over similarities to $M$ fixed text anchors) removes most of the excess gap. The relational target specifies only the geometry that zero-shot scoring later uses, so it constrains the student less than regression does.
For an image $x$, let $t_1,\dots,t_M$ be precomputed, normalized CLIP text embeddings of a broad concept vocabulary, deliberately excluding evaluation class names. Let $f_T(x)$ be the teacher's normalized image embedding and $f_S(x)$ the student's. Let $s(\cdot,\cdot)$ be the student's scoring function. The relational loss is
$$\mathcal{L}(x) = \mathrm{KL}\Big(\mathrm{softmax}_m\big(f_T(x)^\top t_m/\tau\big)\,\Big\|\,\mathrm{softmax}_m\big(s(f_S(x), \tilde t_m)/\tau\big)\Big),$$
where $\tau$ is a temperature and $\tilde t_m$ is the anchor's representation in the student's scoring space. We test two scoring rules. In the first, $s$ is cosine and $\tilde t_m = t_m$. In the second, $s(u,v) = -\|u - v\|_1$ and $\tilde t_m = A t_m$, where $A$ is a matrix learned jointly with the student. Because $A$ is applied to text embeddings, $A t_m$ can be precomputed offline and stored. At deployment, classifying an image then costs one adder forward pass plus $K$ Manhattan distances to stored class prototypes, where $K$ is the number of classes. No multiplications occur apart from normalization layers.
Research Program
The work has three parts. First, we build two students. One is an adder ResNet-50 stabilized with adaptive weight normalization (Towards Stable and Robust AdderNets), which added learnable per-channel scale and shift so adder training stops oscillating. The other is an Adder DeiT-Small using the identity trick of Adder Attention for Vision Transformer. That paper replaced query-key dot products with negative Manhattan distance and added the identity matrix to prevent attention from collapsing to low rank. Both papers supply the method, and neither evaluated contrastive or zero-shot settings. Each adder student has a multiplicative twin with identical depth, width, data, and step count.
Second, we distill with three objectives. Embedding regression serves as the naive target. A kernel-matched feature loss adapts Kernel Based Progressive Distillation, which maps teacher features through a Gaussian kernel and student features through a Laplacian kernel so their different weight statistics can be compared. That method trained teacher and student jointly from scratch, whereas here the teacher is frozen. The third objective is the anchor-relational loss above. We cross each objective with the cosine and Manhattan scoring heads.
Third, we analyze where the gap comes from. We measure how the student's image-to-anchor similarity rankings agree with the teacher's on seen versus unseen concepts. We check whether the Laplacian-shaped, center-clustered structure reported in AdderNet predicts which unseen classes fail.
CLIP (Learning Transferable Visual Models From Natural Language Supervision) motivates the whole setting: it showed that prompt-based classification transfers without target labels. It also constrains the design. Its image and text embeddings share a cosine space, so any student must be judged on whether it preserves that space's structure, not just its nearest neighbors. We do not retrain the text encoder, and we do not attempt adder contrastive pre-training from scratch.
Baselines
The upper reference is the frozen CLIP teacher's own zero-shot accuracy. The file reports 76.2% zero-shot on ImageNet for CLIP's best model (Learning Transferable Visual Models From Natural Language Supervision). Our teacher may be smaller, so we re-measure it under our prompts, and all gaps are computed against that re-measured number.
The strongest competitor on the efficiency question is post-training quantization of the teacher's image encoder to INT8 and INT4. It needs no training and runs on commodity hardware. No reported result was found on Lacuna for quantized CLIP zero-shot accuracy, so we run it ourselves on the same evaluation sets and prompts. The adder student beats it only if it matches or exceeds quantized-teacher accuracy at lower estimated arithmetic energy. We estimate energy by counting operations at 3.7 pJ per 32-bit multiply and 0.9 pJ per add (Adder Attention for Vision Transformer), and we report integer-width variants separately.
The strongest competitor on the accuracy question is the matched multiplicative student trained with the same objective. It isolates the cost of the adder operation itself. For supervised reference points, the file gives 74.9% versus 76.2% top-1 for adder versus multiplicative ResNet-50, and 78.3% versus 79.9% for Adder versus standard DeiT-Small. These make the natural yardstick for "no excess gap".
The "do nothing" reference is the adder student distilled by embedding regression with a cosine head. This is what anyone would try first. The ablations isolate the proposal's key ingredients. We swap relational for regression loss while holding the head fixed, and we swap Manhattan for cosine head while holding the loss fixed. A binarized student run through the same pipeline marks the floor. Its supervised ImageNet accuracy of 55.8% (AdderNet) suggests it will be weak. All students get the same images, the same number of optimizer steps, and a fixed tuning budget: the same number of learning-rate and temperature trials, chosen on a validation split drawn from seen classes only.
Why not just…?
Why not just quantize CLIP and skip adder networks? This is the real threat. If INT4 CLIP keeps nearly all zero-shot accuracy, the remaining argument for adders is multiplier-free silicon. That argument rests on hardware figures, such as up to 15× area and power reductions for sum-of-absolute-differences circuits (Clustering Learning for Robotic Vision), not on anything measured here. We will report quantization side by side. If it matches the adder student in accuracy, the claim narrows to deployment targets without integer multipliers. We will also test quantizing the adder student itself, following Q-A2NN, since the two savings can compound.
Why not just distill into a small multiplicative network? That network is exactly our matched twin. If the twin and the adder student show equal excess gaps, the geometric concern is empty. We would then report that the adder operation costs zero-shot transfer nothing beyond its supervised cost, which is a useful result in its own right.
Why not just keep the cosine head and plain regression, and let a longer distillation run fix things? A longer run may shrink the gap on seen concepts while leaving unseen-concept alignment untouched. The seen/unseen class split described below measures precisely this. We give the naive recipe the same step budget and also a doubled budget, so any remaining excess gap cannot be blamed on training length.
Why not train an adder CLIP from scratch? The compute cost is prohibitive, and the result would mix two things: whether adders can learn alignment and whether they can inherit it. Inheritance is what edge deployment needs.
Proposed Experiments
We distill on unlabeled ImageNet training images drawn from only 500 of the 1,000 classes. Zero-shot evaluation then covers three groups. The first is the 500 held-out ImageNet classes, whose images and names the student never saw. The second is CIFAR-10 and CIFAR-100, which differ in resolution and are prompted by class name. The third is ImageNet classes that were seen during distillation. The anchor vocabulary excludes all evaluation class names. The decisive statistic is the excess zero-shot gap: the teacher-to-student drop on held-out classes minus the drop on seen classes, compared between adder and multiplicative students.
We expect three results. With regression plus a cosine head, the adder excess gap should exceed the multiplicative one. The relational loss should shrink it more for adders than for multiplicative students. The Manhattan head with learned prototype mapping $A$ should cost at most about a point relative to a cosine head under the relational loss, which would make the deployment path multiplication-free.
As a secondary test we probe Gap 3 from the file. Adaptive-weight-normalized adder networks held 59.74% under PGD attack on CIFAR-10, a standard gradient-based adversarial attack, while standard CNNs fell to near 0% (Towards Stable and Robust AdderNets). We evaluate all students zero-shot on corrupted and blurred variants of the CIFAR and ImageNet validation images that we generate. This tests whether the robustness to high-frequency attacks carries over to natural shift, or whether low-frequency changes hurt adder students more.
The proposal fails cleanly in either of two ways. If the adder excess gap under naive distillation is indistinguishable from the multiplicative one across three seeds, the geometric hypothesis is wrong. If quantized CLIP dominates the adder student in accuracy at comparable estimated energy, the practical case collapses outside multiplier-free hardware. Either outcome is informative. We will report both, together with operation counts, following the energy accounting used for Adder DeiT-Base (32.9 BpJ versus 80.7 BpJ).
References
- AdderNet: Do We Really Need Multiplications in Deep Learning?
- Kernel Based Progressive Distillation for Adder Neural Networks
- Towards Stable and Robust AdderNets
- Adder Attention for Vision Transformer
- Learning Transferable Visual Models From Natural Language Supervision
- Clustering Learning for Robotic Vision
- Q-A2NN: Quantized All-Adder Neural Networks for Onboard Remote Sensing Scene Classification