Language models trained only on text are bad at "embodied" facts, the properties you learn by handling objects rather than reading about them, such as weight, hardness or temperature. Two careful studies disagree about whether vision fixes this. One finds that the text side of an image-text model like BLIP knows embodied properties far better than a 175-billion-parameter text model, and that distilling this knowledge into a small language model is worth a 134-fold increase in model size (Can Language Models Understand Physical Concepts?). The other finds that vision-language chat models gain only 2.32 points over their text-only twins on embodied questions, and that plain BERT beats them on sensory word meaning (Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?).
We propose that both results are correct, and that the disagreement comes from two hidden variables. The first is which tier of difficulty each benchmark tests: naming one object's property, comparing two objects, or predicting a physical consequence. The second is where visual alignment changes the model. In a contrastively trained text encoder, every text representation is reshaped by image matching. In a chat-style vision-language model, the language backbone is mostly left alone, and it receives visual information only when an image is actually supplied. We will build linked, tiered question chains over the same objects, locate the tier and the representation site where visual alignment carries embodied knowledge, and then test a targeted fix. That fix distills only the ordering of objects along each physical axis from the aligned encoder into a language model. If it works, it should transfer to objects the model never saw labeled.
Core Claim
Our claim is that the useful embodied signal in vision-language representations is relational. Image-text alignment orders objects along properties that change how things look: big things tend to be heavy, glowing things hot, shiny metal hard. That ordering shows up when a question asks for a comparison between two objects. It is invisible to benchmarks that test single-word sensory meaning or non-visual senses, and it does not by itself support multi-step physical prediction.
This makes three predictions that can each fail. Aligned text encoders should beat text-only models mainly at the comparison tier, and only for properties that co-vary with appearance. Chat-style vision-language models queried without an image should behave like their text-only backbones at every tier. No representation-alignment method should help much at the consequence tier unless the comparison tier is already correct. The method contribution follows from the first prediction. Li et al. use maximum mean discrepancy, which matches the overall distribution of student and teacher vectors. If the signal is relational, a loss that matches pairwise structure among object representations should transfer more of it.
Research Program
The first component is a tiered benchmark of linked chains. The hierarchical-evaluation literature, from CLEVRER to PhD and MVU-Eval, scores each tier separately. The research file also lists modeling how errors propagate between tiers as an open problem. We therefore build chains in which the same object pair appears at three tiers. At tier 1 the model judges an absolute property, for example whether a bowling ball is heavy. At tier 2 it compares the pair: which is heavier, a bowling ball or a tennis ball. At tier 3 it predicts a consequence of that comparison, for example which ball, rolled at the same speed, pushes a pin farther.
Object pairs and property axes are seeded from VEC, which covers mass, temperature, hardness, size and material, and from PerceptualQA's tactile and visual items. Tier-3 templates follow UTOPIA's six physical domains of motion, friction, free fall, projection, collision and incline (Mind's Eye), restated with everyday objects. Labels come from reference physical values where they exist and from three-annotator agreement otherwise.
Each axis is tagged as visually co-varying (size, material, mass for ordinary objects) or not (temperature of indistinguishable liquids, smell, interoception). Each item is also tagged with the corpus frequency of its words. Yang et al. show that sensory benchmarks are confounded by word frequency and spelling, so every analysis is stratified by frequency.
The second component is a probe study on matched model pairs. We compare three kinds of representation, all queried without images. The first kind is text-only models: OPT at several sizes, BERT and Mistral. The second is the text towers of contrastive image-text models, CLIP and BLIP. The third is chat-style vision-language models paired with their own backbone, LLaVA-Mistral against Mistral, which is the pairing Yang et al. used.
Each model is evaluated two ways. The first is direct answering where the model can generate. The second is a frozen linear probe, a small linear classifier trained on top of the model's internal vectors. Probe items are split by object, so a probe never sees test objects. We report accuracy per tier and per axis type. We also report conditional accuracy, meaning tier-2 accuracy given that tier 1 was answered correctly on the same objects, and likewise for tier 3 given tier 2. These conditional numbers measure error propagation directly rather than inferring it.
The third component is relational distillation. The teacher is the aligned text encoder; the student is a 1.3B language model, matching Li et al.'s setting. For a vocabulary of object names, let $t_i$ be the teacher's vector for object $i$ and $h_i$ the student's vector at a chosen layer. We train the student with its usual language-modeling loss plus
$$ L_{\text{rel}} = \sum_{i,j} \big( \cos(h_i, h_j) - \cos(t_i, t_j) \big)^2 , $$
which asks the student to reproduce the teacher's pattern of object-to-object similarities without forcing its vectors into the teacher's space. This is the graph-of-similarities view that EGO uses for video-text fusion, there as a robust alternative to aligning raw features. Matching relations instead of raw vectors also avoids the trade-off reported by Extending Multi-modal Contrastive Representations: a single mapping into a fixed base space sacrifices either fine-grained structure within a modality or alignment between modalities.
A variant restricts the similarity matrix to the axes that showed a tier-2 gain in the probe study. This tests whether targeting what the diagnosis found matters. Distillation uses only object names and caption text, never benchmark questions, so any gain on held-out objects cannot come from memorized answers.
Baselines
The naive reference point is the undistilled 1.3B student, together with the text-only scaling curve up to OPT-175B. Li et al. report that OPT-175B reaches 55.79% average accuracy on VEC's embodied concepts and sits near 50% on mass. Any method that fails to beat the 1.3B student at tier 2 on visually co-varying axes has learned nothing from vision.
The strongest representation-transfer baseline is Li et al.'s maximum mean discrepancy distillation from the same teacher into the same student. It is the only existing method that moves knowledge from an aligned space into a language model. Its teacher, BLIP-base, scores 71.95% on VEC's embodied concepts. We rerun it with identical teacher, student, object vocabulary, caption text, training steps and a matched hyperparameter search of the same number of trials. Beating it means higher tier-2 accuracy on held-out objects with non-overlapping 95% bootstrap intervals, and no loss at tier 1.
The strongest supervised baseline is direct fine-tuning on embodied questions. Yang et al. report that fine-tuning GPT-4o-Mini raises PerceptualQA accuracy from 58.21% to 79.29%, against 85–90% for humans. We fine-tune the same 1.3B student on PerceptualQA training items plus tier-1 and tier-2 chain items for training objects, using LoRA and, separately, attention-LayerNorm tuning (Tuning LayerNorm in Attention). LayerNorm tuning is included because it is the more effective low-parameter adapter in that paper. Both fine-tuned models get the same compute as distillation. To beat this baseline, distillation must match it on training objects and exceed it on held-out objects and at tier 3.
For tier 3 we include the Mind's Eye simulator pipeline as an upper reference on items that can be simulated. For UTOPIA, Mind's Eye reports absolute gains of 27.9% zero-shot and 46.0% few-shot, and nearly 99% few-shot with Instruct-GPT. No reported result was found on Lacuna for our everyday-object tier-3 chains, so we will report it ourselves.
Two ablations isolate the key ingredient. The first replaces the image-aligned teacher with a text-only encoder of similar size, which tests whether the gain comes from visual alignment at all. The second shuffles teacher similarities within each axis, which keeps the loss's form but destroys the ordering.
Why not just…?
Why not just fine-tune on embodied QA, since that already gains 21 points? This is the most serious threat. Yang et al.'s 58.21% to 79.29% result shows that labeled questions work. The open question is whether fine-tuning teaches properties or teaches answers for the objects it saw. Our held-out-object split and our tier-3 chains, which contain no training-format questions, settle this. If fine-tuning transfers to unseen objects and to consequence questions as well as distillation does, then relational distillation is unnecessary and we will report that. Its remaining case would then be coverage: caption text is cheap, labeled embodied questions are not.
Why not just reuse MMD distillation from Li et al.? It is the direct predecessor, and it may work as well. Matching distributions of vectors does not specifically preserve which object outranks which. The head-to-head comparison at tier 2 on held-out objects, plus the shuffled-similarity ablation, shows whether the relational structure is what carries the gain. If MMD ties, the contribution reduces to the diagnosis, which still explains the Li–Yang disagreement.
Why not just call a physics simulator? For tier-3 rigid-body questions it will likely win; Mind's Eye nearly saturates UTOPIA. It cannot answer tier-1 or tier-2 questions about objects whose parameters the model gets wrong, because the code it generates inherits those errors. It also does not cover touch, temperature or interoception. Our conditional-accuracy analysis measures exactly this: the simulator's tier-3 accuracy given a wrong tier-2 belief.
Why not just scale the text model? Li et al. show near-zero scaling gains on mass, temperature and hardness up to 175B parameters. We include the OPT size curve so this baseline is measured on our chains rather than assumed.
Proposed Experiments and Decisive Checks
The first check tests the reconciliation. We fit a model with tier × representation-kind × axis-type interaction terms to per-item correctness. The claim survives if aligned text towers beat text-only models mainly at tier 2 on visually co-varying axes, with the advantage shrinking on non-visual axes. A second condition is that LLaVA-Mistral without images stays within a few points of Mistral at every tier, which would reproduce Yang et al.'s 2.32-point gap as a special case. The claim fails if the aligned advantage is uniform across tiers and axis types, or vanishes once results are stratified by word frequency.
The second check tests error propagation. Tier-3 accuracy conditional on a correct tier 2 should be far above unconditional tier-3 accuracy for every model. If it is not, tier-3 failure is independent of property knowledge, and representation transfer cannot fix it.
The third check tests the method. On held-out objects, relational distillation should beat both MMD distillation and fine-tuning at tier 2, and its tier-3 gain should be roughly what tier 2's improvement predicts through the conditional rates. The axis-targeted variant should beat the untargeted one; otherwise the diagnosis added nothing to the intervention.
As a secondary check, we report the distilled student on SensoryVec and PerceptualQA's non-visual tasks. We expect no gain there. A gain would undercut the claim that the transferred signal is specifically visual ordering.
References
- Can Language Models Understand Physical Concepts?
- Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models?
- CLEVRER: Collision Events for Video Representation and Reasoning
- PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset
- MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
- Mind's Eye: Grounded Language Model Reasoning through Simulation
- Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion
- Extending Multi-modal Contrastive Representations
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning