"Most of the cubes are red" is a claim about objects. It is true when red cubes outnumber the other cubes, however much of the picture each cube covers. Formal semantics has long known that quantifiers like "most" cannot be written in classical first-order logic, because they compare the sizes of two sets (Henkin semantics for reasoning with natural language; Problems in Logical Form). Theory has also shown that transformers are exactly as expressive as first-order logic extended with a "majority" quantifier, and that attention behaves like weighted majority voting over tokens (Transformers Implement First-Order Logic with Majority Quantifiers). So a transformer can in principle evaluate "most", but only over whatever its tokens stand for. In today's vision-language models, a small learned layer (the projector) turns each square image patch into one token for the language model. The tokens therefore stand for patches, not objects. A majority vote over patches measures how much area is red, which is a different question from how many cubes are red.
This note proposes to test that mismatch and then fix it. We will build scenes in which area and object count point in opposite directions, for example three large blue cubes beside five tiny red ones, and ask whether vision-language models judge "most" by count or by area. We then compare the standard patch projector against a projector that pools patches into one token per object, with the language model, data and training budget held fixed. If the hypothesis holds, we get a concrete account of one known failure of these models on quantifiers and a cheap change to the projector that targets it.
Core Claim
The claim has two parts. The first part is a prediction. When a projector maps patches to tokens and objects differ in size, a model's yes/no judgment on proportional quantifiers ("most", "few", "more than half") should track the area proportion of the target property rather than the count proportion. The reason is that the cheapest computation a transformer has available, a normalized vote over tokens, computes area under patch tokenization. The second part is a design consequence. Pooling patches into object-level tokens before projection makes the cheap vote compute count. It should improve conflict-scene accuracy and, above all, generalization when object sizes shift away from training. Fine-tuning on decorrelated data alone should help less.
We measure which signal a model uses by fitting, for each model, a logistic model of its "yes" probability on a "most A are B" query:
$$ P(\text{yes}) = \sigma\big(\beta_c\,(c - \tfrac12) + \beta_a\,(a - \tfrac12) + \beta_0\big) $$
Here $\sigma$ is the logistic function, $c$ is the fraction of A-objects that are B, $a$ is the fraction of A-area that is B, and $\beta_c$, $\beta_a$, $\beta_0$ are fitted coefficients. A model that interprets "most" correctly has $\beta_a \approx 0$. The prediction is that patch-token models show $\beta_a \gg \beta_c$ unless trained otherwise.
This framing differs from the closest prior work. Be Precise or Fuzzy learns mappings from quantifier words to whole-image vectors on patchwork scenes. It finds that cosine similarity suits proportional quantifiers and dot product suits exact numbers, but it never asks what the units of quantification are. Quantifiers in a Multimodal World grounds nine quantifiers in a small audio-visual hub network, again without autoregressive language models or a separation of area from count. Both motivate the question; neither tests individuation.
Research Program
The program has three stages.
Stage one: diagnostic data. We render controlled scenes with the generator behind CLEVR. For each query we sample object sizes so that count proportion and area proportion are independently controlled. Some scenes have the two agreeing, some conflicting, and some with area fixed at 0.5 while count varies (and vice versa). Query templates cover "most", "few", "more A than B", and exact cardinals as a control. We also re-render the Be Precise or Fuzzy patchwork setup with variable patch sizes, so that the result connects to the earlier cardinal/quantifier findings.
Stage two: mechanistic localization. We ask where the area/count confusion enters. Interpreting CLIP's Image Representation via Text-Based Decomposition found "counting heads" in the CLIP vision encoder. We regress their outputs on count versus area across our scenes to see whether the encoder itself already conflates the two. We then apply LatentLens, which reads visual tokens against a pool of contextualized text representations, to see whether count information survives the projector into the language model's layers. This separates three possibilities: the encoder conflates area with count, the projector destroys a correct encoder signal, or the language model's voting computes the wrong quantity.
Stage three: entity-level projection. We replace the patch-to-token map with a step that groups patch features into segments and averages each segment into one token before the same MLP projector. Every other part of the model stays fixed. We test three grouping sources:
- ground-truth masks on CLEVR, as an oracle;
- clustering of vision-encoder patch features;
- the encoder's own zero-shot localization, which Gandelsman et al. report at 54.50 mIoU on ImageNet-segmentation.
The projector is trained in isolation with the encoder and language model frozen, the protocol recommended by Scalable Vision Language Model Training via High Quality Data Curation. Because object-level tokens make the attention vote proportion-like and therefore scale-invariant, we expect exact cardinals to still need magnitude information, as Be Precise or Fuzzy found. We therefore report cardinals separately and do not claim the fix covers them.
Baselines
Language prior only. The naive reference is a text-only answerer that sees the query but not the image. On NLVR2 we also report the image-only blind majority-label baseline, which reached 83.53% dev accuracy according to NLVR2 Visual Bias Analysis. That number shows why NLVR2 results must use the balanced protocol from the same paper.
Standard patch projector. The main end-to-end competitor is a LLaVA-style two-layer MLP projector of the kind used in MiniGPT-v2 and Video-LLaVA. It uses the same vision encoder, language model, training data and step budget as our variant.
Deep Pre-Alignment. The strongest existing projector design is Deep Pre-Alignment. It passes vision-encoder tokens through a small language model before projection and reports a 1.9-point average multimodal gain at 4B parameters. It also reports MATH-500 retention of 54.2 against 36.4 for the standard baseline. It is the right competitor because its deeper preprocessing could, in principle, learn to individuate objects without explicit grouping. No reported result was found on Lacuna for Deep Pre-Alignment on quantifier or counting tasks. We will run it with matched compute and tuning sweeps.
Pipeline system. The symbolic upper reference detects objects with ground-truth or detector masks, counts them, and evaluates the quantifier as an explicit set comparison. This is the visual analogue of the comparison axioms that Logical Inferences with Comparatives and Generalized Quantifiers feed to a theorem prover, reaching 84% on MED.
Ablations. Two ablations isolate the key ingredient. The first, random grouping, keeps the token count and segment sizes but assigns patches to groups at random. This separates individuation from mere token reduction. The second, area-weighted entity tokens, duplicates each object token in proportion to its size. This should reintroduce the area bias if the mechanism is the one we claim.
What counts as beating a baseline. Beating the standard projector means:
- higher accuracy on conflict scenes;
- $\beta_a$ not significantly different from zero;
- held-out-size accuracy within a few points of in-distribution accuracy.
Each of these must hold over at least three seeds. Against Deep Pre-Alignment, entity pooling must match or exceed it on conflict scenes at lower cost.
Why not just…?
Why not just fine-tune the standard model on scenes where area and count disagree? This is the most serious threat, and it is a primary arm of the experiment rather than something we argue away. Our prediction is that it repairs in-distribution conflict accuracy but fails when object sizes shift. With patch tokens, count has to be reconstructed by a size-normalizing computation that the vote does not natively perform. The CLEVR-CoGenT result, where accuracy fell from about 85% to about 51% under swapped attribute combinations (CLEVR), shows that end-to-end training memorizes training correlations. If decorrelated fine-tuning matches entity pooling on held-out sizes, the design claim is refuted and the contribution shrinks to the diagnostic.
Why not just prompt the model to count first, with chain-of-thought? Step-by-step reasoning adds sequential depth beyond the single forward pass analyzed by Merrill & Sabharwal, so it could sidestep the voting bias. However, FRoG shows that even text-only models that compute the right percentage often pick the wrong quantifier. In 8 of 15 model families, larger models did worse, so more reasoning is not a dependable fix. We run count-then-answer prompting on every model and report whether the counts it states are themselves area-biased.
Why not just use a detector and a counting program? On CLEVR this pipeline will likely win, and we treat it as an upper bound, not a rival. The question is what end-to-end models do. They are the systems deployed on natural images, where pipelines must commit to brittle detections; NLVR2 statements mix quantifiers with open-ended relations. Entity pooling keeps individuation as a soft, trainable step inside the model. If the pipeline also beats us on balanced NLVR2, we report that plainly.
Why not just use a dot-product versus cosine routed projector, as Be Precise or Fuzzy suggests? Changing the similarity metric does not change what is being counted. A cosine-style proportion over patch tokens is still an area proportion. The area-weighted ablation tests this directly.
Proposed Experiments
Decisive experiment. We use the CLEVR-generated area/count conflict set. Training scenes have object sizes from one range and test scenes from a disjoint range. We report conflict accuracy and the fitted coefficients $\beta_c$ and $\beta_a$ for six models:
- an off-the-shelf patch-projector vision-language model;
- the matched standard projector;
- Deep Pre-Alignment;
- entity pooling under each of the three grouping sources;
- both ablations;
- the decorrelated fine-tune.
No reported result was found on Lacuna for modern vision-language models on any area-controlled quantifier test, so stage one is itself a new measurement.
Transfer to existing benchmarks. We compare against the reported numbers on four benchmarks:
- On re-rendered patchwork scenes, the reference is the cosine network's 0.68 mAP on "few" from Be Precise or Fuzzy.
- On the Pezzelle et al. quantifier distributions, the hub model reaches Pearson's r of 0.86.
- On balanced NLVR2, LXMERT reaches 74.0% under the balanced protocol (NLVR2 Visual Bias Analysis), against human accuracy of about 96% (A Corpus for Reasoning about Natural Language Grounded in Photographs).
- On NLVR2 we additionally report the subset of sentences containing quantifiers or comparisons.
Forgetting check. Deep Pre-Alignment showed that projector tuning damages text-only reasoning. We therefore evaluate the language model before and after multimodal training on three text-only sets: MED, FraCaS's quantifier section, and FRoG. This tests whether any visual gain comes at the cost of the text-only quantifier reasoning measured by Haruta et al. and FRoG.
Mechanistic check. We repeat the counting-head and LatentLens probes on the entity-pooled model. Count information should become linearly readable from the language model's early layers, rather than only from the vision encoder.
References
- Henkin semantics for reasoning with natural language
- Problems in Logical Form
- Transformers Implement First-Order Logic with Majority Quantifiers
- Be Precise or Fuzzy: Learning the Meaning of Cardinals and Quantifiers from Vision
- Quantifiers in a Multimodal World: Hallucinating Vision with Language and Sound
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- Interpreting CLIP's Image Representation via Text-Based Decomposition
- LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs
- Scalable Vision Language Model Training via High Quality Data Curation
- NLVR2 Visual Bias Analysis
- MiniGPT-v2: Large Language Model as a Unified Interface for Vision-Language Multi-task Learning
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
- Deep Pre-Alignment for VLMs
- Logical Inferences with Comparatives and Generalized Quantifiers
- FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Models
- A Corpus for Reasoning about Natural Language Grounded in Photographs