Vision-language models such as CLIP score an image against a caption by measuring the angle between an image vector and a text vector. These models are bad at counting. Zero-shot CLIP picks the right count caption on only about a third of the CountBench images, and every published fix stops at ten objects. This note proposes to locate the failure. It may be that the image encoder never represents "how many." It may be that the text encoder never orders number words. Or both sides may carry an ordered magnitude axis, and the two axes simply point in unrelated directions across the gulf that separates image vectors from text vectors in these models. That gulf is called the modality gap. If the third account holds, a small rotation restricted to a few dimensions should repair counting, extend past ten without paired high-count captions, and leave everything else the model knows untouched.
The opportunity comes from putting two findings side by side. Language models spontaneously build low-dimensional, monotonic "number lines" in their hidden states. Vision models are widely believed not to, but that belief rests on probing studies with serious statistical problems and on area and density confounds. Nobody has measured the angle between a visual magnitude axis and a textual one inside the same shared embedding space. That angle is the quantity this project measures and then corrects.
Core Claim
We hypothesize that, for counts well beyond the small-number range, CLIP's image embeddings contain a magnitude direction that is decodable after controlling for confounds. We further hypothesize that its text embeddings contain an ordered count direction that generalizes across nouns, and that zero-shot counting fails mainly because these two directions are misaligned. We do not think the failure comes from missing information, and we do not think it comes from the overall size of the modality gap.
The proposed intervention, which we call a number-axis bridge, is a learned map that touches only a $k$-dimensional count subspace of the image embedding. It is fitted on synthetic, confound-controlled images. Because it maps onto a continuous text-side axis rather than onto a fixed set of count classes, it should transfer to natural images and extrapolate to counts never shown during fitting.
Prior Work and Where This Sits
On the language side, two papers motivate the hypothesis that text carries a usable line. Language Models Encode the Value of Numbers Linearly reads number values out of LLM hidden states with linear probes reaching $R^2 > 0.95$. The same paper shows exact reconstruction stays below 50% approximate accuracy, so these lines are coarse. Monotonic Representation of Numeric Attributes in Language Models finds that 2–6 dimensions found by partial least squares (a regression that picks directions jointly predictive of the target) carry most of the predictive power, and that patching along them steers outputs monotonically. We borrow that subspace-finding method and apply it to both CLIP encoders rather than to an LLM. Behavioural studies such as Cognitive Effects in Large Language Models report distance and ratio effects. These suggest the text line may be log-compressed, which we test directly.
On the vision side, the literature constrains the design. On Numerosity of Deep Neural Networks shows that "number neurons" in ImageNet networks were a small-sample artifact: untrained networks showed more of them, and they vanished at 80 images per condition. Cognitive Deficit of Deep Learning in Numerosity shows that convolutional networks track pixel area and density rather than count. Our measurement therefore uses their area-matched and convex-hull-matched stimulus controls, large per-condition sample sizes, and an untrained-encoder control. We treat a null visual result as a real possibility.
On the modality-gap side, Mind the Gap explains the gap as a cone effect that appears at initialization and is preserved by contrastive training. It shows that shifting the gap is a tunable knob, moving CIFAR-10 zero-shot accuracy from 0.9013 to 0.9081. That work treats the gap as a single global offset. We ask about the relative orientation of one semantic axis across it, which a global shift cannot fix.
The closest prior work is the cross-modal counting literature. Teaching CLIP to Count to Ten fine-tunes CLIP with counterfactual count captions and reaches 75.93% on CountBench, but collapses above ten. Improving CLIP Counting Accuracy via Parameter-Efficient Fine-Tuning learns nine discrete counting vectors added to text embeddings. It never touches the image side and cannot represent counts past ten. Probing Representations of Numbers in Vision and Language Models finds that language features dominate visual ones above four in ViLBERT, but never measures cross-modal geometry. Evaluating and Mitigating Number Hallucinations tunes a projection connector for behavioural consistency without inspecting it.
Our proposal differs from all of these in three ways. It treats count as a continuous axis on both sides. It edits the image side. And it confines the edit to a subspace measured in advance.
Research Program
The first stage is measurement. We render synthetic scenes containing 1 to 50 objects, in the style of the GrayCount250 line art and dot patterns. They come in three regimes: unconstrained; total area and density matched across counts; and convex hull matched across counts. Each condition gets hundreds of samples and many object categories. We encode these with CLIP ViT-B/32 and ViT-L/14, and also with a randomly initialized copy of each.
On the image side, we fit partial-least-squares count subspaces with held-out object categories and held-out counts. On the text side, we fit the same kind of subspace on embeddings of templated captions ("a photo of N cats") for N from 1 to 100 over many nouns. For each side we report held-out $R^2$, Spearman correlation, and whether linear or log count fits better. Across sides we report the principal angles between the two subspaces.
The visual claim survives only if the trained encoder beats the untrained one on the area-matched set. We also need a sanity check on the text side. If the CLIP text encoder turns out to have no ordered line past ten, the bridge target becomes the text embedding of the nearest well-ordered count. The text side then becomes a reported finding in its own right rather than something we quietly patch.
The second stage is the bridge. Let $v$ be an image embedding. Let $U_I$ be the orthonormal basis of the image count subspace ($k$ columns) and $U_T$ the basis of the text count subspace. The bridged embedding is
$$v' = v - U_I U_I^\top v + U_T\, W\, U_I^\top v ,$$
where $W$ is a $k \times k$ matrix. Applied to each image's own embedding, the update replaces that image's coordinates inside the image count subspace with mapped coordinates inside the text count subspace. Everything orthogonal to $U_I$ is left as it was.
$W$ is fit on synthetic images with counts 1–20 only, so that each image's mapped coordinates match the text-side coordinates of its correct count caption. We compare an orthogonal (Procrustes) $W$ with an unconstrained one; the orthogonal version is the "just rotated" hypothesis made literal. With $k$ around 2–6, as the Wikidata results suggest, there are only a few dozen parameters.
The third stage is evaluation. We test zero-shot counting on natural images, extrapolation to counts 21–50, and retention of general zero-shot ability.
Baselines
The naive reference point is frozen zero-shot CLIP, scored by choosing the highest-similarity count caption. Reported numbers are 31.67% on CountBench for CLIP-B/32 (Teaching CLIP to Count to Ten) and 33.62% on DiverseCount for CLIP-L/14 (Improving CLIP Counting Accuracy). We rerun both under our prompt templates so the comparison uses identical scoring.
The strongest existing method within the 1–10 range is counting-contrastive fine-tuning from Teaching CLIP to Count to Ten, at 75.93% CountBench accuracy for CLIP-B/32. For fairness we retrain it on exactly our synthetic 1–20 images plus its own curated data, with the same compute and the same hyperparameter search budget as the bridge.
Beating it means two things. First, it means matching it within a few points on CountBench while staying within one point of base CIFAR-100 zero-shot accuracy, which is about 48% according to Improving CLIP Counting Accuracy, where full fine-tuning fell to about 35%. Second, it means clearly exceeding it on held-out counts 21–50. No reported result was found on Lacuna for CLIP zero-shot counting above ten. GrayCount250's published numbers concern diffusion generation, not CLIP recognition, so that comparison is entirely our own measurement.
The strongest parameter-efficient competitor is the orthogonal text-side counting vectors of Improving CLIP Counting Accuracy, at 41.34% on DiverseCount for CLIP-L/14. We extend it to vectors $\Delta_2$ through $\Delta_{20}$ and train it on the same data. This tests directly whether editing the image side matters.
The gap-shift baseline from Mind the Gap translates image embeddings by a tuned multiple of the mean modality difference. It is tuned on the same validation split.
We also run three ablations that isolate the key ingredients. A full-rank Procrustes map applied to the whole embedding tests whether confining the edit to a subspace protects general ability. A bridge fit on unconstrained rather than area-matched images tests whether confound control matters. A bridge fit on counts 1–10 only tests whether extrapolation comes from the continuous axis or from training coverage.
Why not just…?
Why not just train a count regressor on frozen image features? For counting accuracy alone, this is a real threat. A head trained with the same synthetic data may match or beat the bridge, and we will report it as an upper reference. But it outputs a number, not an image embedding that a text query can score. It cannot answer "is this five apples or six pears" without a separate detector, and it cannot feed text-conditioned retrieval. The bridge is worth having only if it comes close to the regressor while keeping the text interface. We will report the size of that difference honestly.
Why not just extend counting-contrastive fine-tuning with synthetic high-count images? This may well work for the counts it is trained on. The evidence against it is that it updates all weights, and full fine-tuning on counts erased about 13 points of CIFAR-100 zero-shot accuracy (Improving CLIP Counting Accuracy). As a classification-style objective, it has no reason to place count 37 correctly when it has only seen up to 20. The extrapolation test and the retention test, both run under matched budgets, settle this.
Why not just shift the modality gap? Mind the Gap shows that a global translation helps classification. A translation adds the same vector to every image, however. If the visual and textual count axes are at a large angle, no constant offset can make image coordinates along one axis predict caption ranks along the other. Our principal-angle measurement predicts whether gap shifting can help before we even run it, and the gap-shift baseline checks the prediction.
Why not just rely on scale? Demystifying Numerosity in Diffusion Models found negligible numerosity gains from 2B to 12B parameters. Evaluating and Mitigating Number Hallucinations found 7B to 13B did not remove number hallucination. Comparing B/32 with L/14 in our measurement stage tells us whether the axis angle shrinks with scale. If it does, that is itself a useful finding.
Decisive Checks
Three results would decide the project. First, the trained CLIP image encoder must show a count axis on area-matched and hull-matched stimuli that clearly beats its untrained twin. If it does not, the "rotated" hypothesis fails at its first step, and the paper becomes a confound-controlled negative result extending On Numerosity of Deep Neural Networks to contrastive models.
Second, the angle between the image and text count subspaces must predict zero-shot counting error across models and object categories better than the overall gap distance does.
Third, a bridge with a few dozen parameters, fit only on synthetic 1–20 images, must improve CountBench and DiverseCount above the frozen baseline and above extended text-side counting vectors. It must also keep Spearman correlation high on held-out counts 21–50 and leave CIFAR-100 zero-shot accuracy within a point of the base model.
If all three hold, CLIP's counting failure is largely a problem of orientation across the modality gap, and it can be fixed by geometry rather than by data. If only the first holds, the visual information is present but cannot be reached linearly, which points toward the nonlinear and morphological approaches of Cognitive Deficit of Deep Learning in Numerosity.
References
- Language Models Encode the Value of Numbers Linearly
- Monotonic Representation of Numeric Attributes in Language Models
- Cognitive Effects in Large Language Models
- On Numerosity of Deep Neural Networks
- Cognitive Deficit of Deep Learning in Numerosity
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Teaching CLIP to Count to Ten
- Improving CLIP Counting Accuracy via Parameter-Efficient Fine-Tuning
- Probing Representations of Numbers in Vision and Language Models
- Evaluating and Mitigating Number Hallucinations in Large Vision-Language Models: A Consistency Perspective
- Demystifying Numerosity in Diffusion Models — Limitations and Remedies