Does Vision Fail to Teach Syntax Because It Is Too Coarse?

Children learn grammar from roughly 100 million words, while language models need hundreds of billions of tokens. One popular explanation is that children also see the world, and what they see disambiguates what they hear. A clean test of this idea found a split result. When a small language model was trained on synthetic scenes, images taught it hierarchical grammar rules, such as making a verb agree with the true subject rather than the nearest noun, within 100 training steps. On real photographs with captions, the images helped by less than 1% over a white-noise control (Does Vision Accelerate Hierarchical Generalization in Neural Language Learners?). Stronger vision encoders did not change this. Medical imaging ran into a similar problem and largely solved it. Chest X-rays are diffuse and cluttered, and reports describe small local findings. Global "this image goes with this report" training missed those findings. Methods that align individual report words with individual image patches recovered them (Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning).

The proposal is to test one specific diagnosis: real-image grounding fails to teach syntax because the image signal reaches the language model only as a single summary vector. A disambiguating cue, such as "there are two dogs," then cannot be tied to the particular noun whose number decides agreement. We will import word-to-patch alignment from label-efficient medical pretraining into child-scale multimodal language learning. We will measure whether it brings back on natural images the syntax benefit that vision gives on synthetic ones, and we will include controls that can show the diagnosis is wrong.

Core Claim

Both literatures face the same structure. A target the training stimulus does not determine must be inferred from a weak side channel. In medicine the target is a disease label available for only 1% of images, and the side channel is the paired report. In developmental language learning the target is a grammatical rule that ambiguous sentences cannot settle, and the side channel is the paired scene. In Kuribayashi and Baldwin's synthetic scenes there are few objects, so a pooled image vector already encodes the subject's number. In natural photos the subject is one region among many, and pooling washes it out.

Our claim is that adding a token-level cross-attention loss will make the image measurably useful for hierarchical generalization on natural captions. This is the loss in which each caption word attends over image patches and must be reconstructable from them, and the reverse. The gain should appear exactly on the training items where the subject's referent is visually identifiable. The image encoder stays frozen so that only the alignment granularity changes.

This differs from the closest work in three ways. Kuribayashi and Baldwin diagnose the natural-image failure but test only frozen global encoders feeding a captioning decoder. They try no change to the objective. LexiContrastive Grounding (Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling) aligns early-layer word representations with whole-image DINO features. It improves perplexity by about 5% and word learning, but it explicitly does not target syntax and still aligns words to global image features rather than to regions. MGCA supplies our method but uses it to train an image encoder for X-ray tasks; its text encoder serves only as scaffolding and is never evaluated for language competence. We reverse the direction and ask what fine-grained alignment does for the language side.

Research Program

The base learner follows Kuribayashi and Baldwin's setup: a small autoregressive Transformer of GPT-2-small scale that reads captions conditioned on frozen image features. It is trained on their synthetic environment and on Conceptual Captions subsets in which the agreement ambiguity is present. Their Poverty-of-the-Stimulus benchmark then tests the model, using text only, on whether it generalizes the hierarchical or the linear agreement rule. The test phase never shows images, so any benefit must have been absorbed into the language model's weights.

Onto this learner we add two auxiliary losses borrowed from MGCA. The first is token-wise alignment. Each caption token's middle-layer representation cross-attends over frozen DINO patch features, a symmetric attention runs from patches to tokens, and a local contrastive loss pulls each word toward its attended visual summary. The second is prototype alignment, in which image and caption embeddings are softly clustered into shared groups using the Sinkhorn-Knopp balancing procedure. Its role is to stop the contrastive loss from pushing apart different photos that show the same kind of scene.

We expect token-wise alignment to carry the syntax effect. Prototype alignment addresses false negatives, which matter for representation quality but have no obvious connection to agreement, so it appears mainly as an ablation. We apply the token loss at middle layers because TREEREG finds syntactic features peak there (Sneaking Syntax into Transformer Language Models with Tree Regularization). LCG argues instead for the first layer, so the layer of application is a swept variable rather than a fixed choice.

The analysis has two parts beyond accuracy. First, we hand-annotate about 500 natural training captions for whether the grammatical subject's number is visually determinable in the photo; an offline detector is used only to pre-sort candidates. This splits the data into an "informative" and an "uninformative" subset. Second, we read off the cross-attention maps and check whether subject tokens attend to the subject's region. Together these turn the claim into a mechanism we can check rather than just a score.

Baselines

The do-nothing reference is the text-only learner trained on the same captions with the same steps. Its reported result is hierarchical-generalization F1 of about 40, below chance, in the synthetic setting (Does Vision Accelerate Hierarchical Generalization in Neural Language Learners?).

The paired control is the same architecture fed white-noise images. On natural captions, Kuribayashi and Baldwin report that real images beat white noise by "often less than 1%." Closing this specific gap is the target, so every comparison reports performance minus the white-noise control under matched training.

The strongest existing grounded method for this question is the global-feature multimodal learner from the same paper, rerun with their best frozen encoder. It is reported at about 90 F1 within 100 steps on synthetic data and fails on natural data. We also run LCG's lexical grounding, since it is the strongest published grounding method for small-data language learning. No reported result found on Lacuna for LCG on the Poverty-of-the-Stimulus agreement benchmark or on BLiMP. We also run a global contrastive (CLIP-style) auxiliary loss. This corresponds to MGCA's instance-only variant, whose medical analogue, ConVIRT, reaches 85.9% CheXpert AUROC at 1% labels against MGCA's 88.8%. That is the gap fine-grained alignment bought in the medical setting.

All systems share the frozen DINO encoder, data, token count, optimizer, and an equal hyperparameter search of 20 trials each, and every result is averaged over at least five seeds. The ablations are token alignment alone, prototype alignment alone, both together, and token alignment with images shuffled across captions. The shuffled condition keeps every parameter and the loss while removing true correspondence.

Beating the strongest baseline means two things. On natural captions, the token-aligned model must exceed the global-feature model's margin over white noise by a clear amount; we set the bar at recovering at least a third of the synthetic-environment advantage, with non-overlapping seed intervals. The model must also stay at least as good on the synthetic environment.

Why not just…?

Why not just use a stronger or differently trained image encoder? Kuribayashi and Baldwin already varied encoders, including ViT, Swin, and BEiT at different ImageNet accuracies, and found that encoder quality did not close the natural-image gap. Our design keeps the encoder fixed so that only the alignment granularity varies, which is the variable this negative result leaves untested.

Why not just simplify the captions, as BabyVLM does? BabyVLM rewrites web captions into child-directed language with GPT-4 and gets more efficient compositional learning in compact models. This is a real alternative, because simpler captions may contain fewer distracting nouns and so make global features sufficient. We test it directly by running both the global baseline and our method on rewritten captions. If rewriting alone recovers the synthetic-scale benefit, the granularity diagnosis is weakened and the honest conclusion is that caption complexity, not alignment, was the bottleneck. If our method helps on raw captions and the two effects add, the methods address different causes.

Why not drop vision and add a syntax prior to the text model? TREEREG improves SyntaxGym by up to 9.5 points and halves data needs with a purely textual loss. It could therefore beat any grounded method on raw agreement accuracy. We include it with silver parses as a reference ceiling rather than a competitor, because our question is whether vision can carry the disambiguating signal, not whether syntax can be taught at all. TREEREG also needs an external supervised parser, which children do not have, and the BabyLM Loose-track results show that multimodal fusion in small data is the part that remains poorly understood (Findings of the BabyLM Challenge). We also report token alignment combined with TREEREG to see whether the gains overlap.

Why not use gaze or joint attention as the cue? Sonographer gaze cut labeling needs thirtyfold in ultrasound (Ultrasound Image Representation Learning by Modeling Sonographer Visual Attention), and Kuribayashi and Baldwin speculate that joint attention is what children have. Natural caption corpora carry no gaze, though. Token-to-patch cross-attention is a learned stand-in for "attend to what the word refers to," so a positive result here would also make a gaze study on egocentric video worth its cost.

Proposed Experiments

The primary experiment trains every system on the synthetic environment and on Conceptual Captions at three data sizes. It reports hierarchical-generalization F1 against training steps, always as the margin over the matched white-noise run. Secondary evaluations check whether any gain carries beyond the targeted rule: the agreement and anaphora subsets of BLiMP, where reported references are human 88.6%, GPT-2 83.0%, and the best BabyLM models around 90%, plus the agreement suites of SyntaxGym.

Three checks decide the question. First, the shuffled-image ablation must erase the gain; otherwise the loss is acting as a regularizer, not as grounding. Second, the gain must be concentrated in the visually informative subset of the annotated captions. If both subsets improve equally, the mechanism is not referent binding. If neither improves, the natural-image failure most likely reflects missing information in the photos rather than coarse alignment. That would be a useful negative result, because it would rule out the entire family of alignment fixes. Third, subject-token attention mass on the subject's region should rise during training and predict per-item correctness.

A positive outcome would show that a method developed for 1%-label radiology also explains part of the gap between how children and models learn from scenes. A negative outcome would locate the bottleneck in the data rather than in the learner.