Visual-Word Histograms as a Cross-Lingual Pivot for Abstract Words

Pictures can link words across languages without any parallel text. If you search the web for images of "hond" and "dog", the two image sets look alike, so the words are probably translations. This works well for concrete nouns and fails for abstract words such as "justice", "greed" or "freedom". Images of those words are scattered: a gavel, a protest, a balance scale, a courtroom, a stock photo of a handshake. The standard explanation is that abstract words simply have no consistent visual signal. We propose a different explanation. The signal is there, but it lives in the mixture of image types a word evokes, and the usual way of comparing image sets throws the mixture away. Averaging embeddings into one vector, or matching images one by one, collapses a many-peaked distribution into a point.

The proposal is to represent each word, in each language, as a bag of visual words. A bag of visual words is a histogram: every image retrieved for the word is assigned to entries in a shared, language-independent codebook of image clusters built from a frozen vision encoder. Words are then compared through weighted histograms that emphasize rare, distinctive clusters and discount the generic imagery that abstract queries attract. Because both words are compared entirely on the image side, the method avoids the text-to-image representation gap. It requires no parallel sentences and no shared images across languages. The claim is testable and narrow. Relative to modern continuous-feature baselines, gains should concentrate in the high-dispersion, abstract end of bilingual lexicon benchmarks, should vanish for concrete nouns, and should depend on the discounting of generic clusters.

Core Claim

Visual bilingual lexicon induction means inferring translation pairs by comparing images. Visual Bilingual Lexicon Induction with Transferred ConvNet Features established the recipe: retrieve web images per word in each language, embed them with a CNN, and compare the two sets. That paper motivates the present one in two ways. It measured "image dispersion", the variance of a word's image set, as a proxy for concreteness, and it reported abstract, high-dispersion words as the core failure. It also retired bag-of-visual-words after continuous AlexNet features beat SIFT-based bags by 79% relative in mean reciprocal rank on BERGSMA500. That comparison, however, confounded two things: the quantization step and the weak descriptors being quantized (SIFT, a hand-designed local gradient feature).

Modern codebooks are built over deep embeddings instead. DCBM clusters CLIP embeddings of segmented crops with k-means into discrete "visual concepts" and reaches 82.4% on CUB-200-2011. That removes the confound, and we reuse its codebook construction.

On the language side, Semantic Similarity Computation for Abstract and Concrete Nouns found a different geometry for the two word types. Concrete nouns are best compared by their single closest sense over large neighborhoods of 30–40 words. Abstract nouns are best compared through attributes over small neighborhoods of about 10 words. Our hypothesis transfers that finding to vision. An abstract word's image set is a set of visual attributes, and a histogram with small-support, attribute-style comparison should translate it better than a centroid or a best-match rule.

Concretely, let $h_w \in \mathbb{R}^K$ be the soft-assignment histogram of word $w$'s images over $K$ codewords. Let $N$ be the number of words in both vocabularies, and $\mathrm{df}_k$ the number of words whose image sets use codeword $k$. We score a candidate pair $(w, v)$ as

$$ s(w,v) = \cos\big(\tilde h_w, \tilde h_v\big), \qquad \tilde h_{w,k} = h_{w,k}\,\log\frac{N}{\mathrm{df}_k}, $$

keeping only each word's top-$n$ weighted codewords, with $n$ small, following the attributional finding above. The log term is the standard term-weighting (tf-idf) discount from information retrieval. We expect it to matter specifically for abstract words, whose searches return shared stock imagery.

Research Program

We first build a single codebook from a language-balanced pool of images: whole images plus region crops proposed by Segment Anything, as in DCBM. We embed the pool with a frozen CLIP ViT image encoder and cluster it at $K \in \{256, 1024, 4096\}$. Per-word image sets are retrieved with native-language queries, following Kiela et al. As a second source we use images that co-occur with the word in interleaved documents from PIN-14M, where language coverage allows. This tests whether contextual imagery, not only direct search hits, carries attributional signal. Following Extracting Translation Pairs from Comparable Corpora through Graph-based Label Propagation, we add reverse-translation filtering, which keeps a pair only if the target maps back to the source.

The final system fuses the visual score with a text score. Fusion weight is set by a word's image dispersion, and we compare this against the Weighted Gram Matrix fusion from Multi-Modal Models for Concrete and Abstract Concept Meaning. That fusion first turns each modality into a word-by-word similarity matrix and then combines the matrices by weight.

Two further works shape the design. Unsupervised Bilingual Lexicon Induction from Mono-Lingual Multimodal Data instead learns localized visual features from monolingual caption corpora. It reaches 76.6% P@1 on German-English BERGSMA500, but it needs caption datasets, which are scarce. We need only image search or interleaved web documents. Learning Robust Visual-Semantic Embeddings argues that pairwise matching leaves global distributions misaligned and corrects this with a distribution-matching loss called Maximum Mean Discrepancy (MMD). We take that distributional view at the level of a single word's image set, and we test MMD directly as a competitor.

Hill et al. also act as a constraint. They found that adding perceptual information to abstract verbs consistently degrades representations. Conjuring Semantic Similarity likewise found that diffusion-based visual grounding loses verb and adjective information. We therefore predict no gain on abstract verbs and will report that stratum separately rather than averaging it away.

Baselines

The naive reference points are chance ranking and orthographic similarity. Orthographic similarity (normalized edit distance between the source and target word) matters because cognates inflate results on related pairs such as Dutch-English.

The strongest visual competitor is the Kiela et al. recipe re-run with a modern encoder. It uses the same retrieved images and the same frozen CLIP encoder, scored by AVGMAX (the average over source images of the best-matching target image) and by CNN-MEAN (cosine between set centroids). Kiela et al. showed AVGMAX tracks translation equivalence rather than mere relatedness. With CLIP features it is exactly what a reviewer would propose, and it isolates quantization and weighting as the only difference from our method. The published reference numbers on the file's benchmarks are 76.6% P@1 on German-English BERGSMA500 Chen & Jin and the 79% relative MRR gain over SIFT bags Kiela et al.. For VULIC1000, Kiela et al. report beating linguistic bootstrapping on Dutch-English, but no reported numeric result was found on Lacuna.

The strongest text competitor is a multilingual text encoder (M-BERT or XLM, as evaluated in Multi-SimLex, plus a multilingual CLIP text tower), with mean-centering and all-but-the-top post-processing applied. Multi-SimLex reports that such models often fail to break Spearman $\rho = 0.6$ against a human agreement of 0.74. We also include the text-only label-propagation system of Tamura et al., which reached 10.8% Top-1 with a large seed lexicon and 4.1% with a small one on the English-Japanese patent corpus.

All visual methods share the same images, the same encoder and the same hyperparameter search budget, tuned on a held-out split drawn equally from concrete and abstract words. Beating AVGMAX means a statistically significant gain in P@1 and MRR in the top dispersion tercile, with no significant loss in the bottom tercile. Beating the text encoder alone is not required. The relevant target is that fusion beats text alone on abstract strata.

The ablations that isolate the key ingredient are histograms without idf, a random codebook of the same size, varying $n$ and $K$, and search-only versus PIN-derived images.

Why not just…?

Why not just use a strong multilingual text encoder? This is the most serious threat to any claim that visual-only scores are better. We do not make that claim. Multi-SimLex shows these encoders degrade most on abstract words, verbs and low-resource pairs, which is exactly where a language-independent visual pivot could add information. The test is whether dispersion-gated fusion improves the text encoder on high-abstractness Multi-SimLex cross-lingual pairs. If it does not, the visual pivot has no practical value there and we will say so.

Why not just use CLIP embeddings with AVGMAX, as Kiela did? This is plausible, since CLIP is far stronger than AlexNet. But AVGMAX rewards a single lucky best-matching image, and a centroid averages distinct visual modes into a vector that matches none of them. Both behaviors are what Kiela et al. saw break on dispersed image sets. The stratified comparison with identical images settles the question directly.

Why not just compare the continuous image sets as distributions, with MMD or optimal transport, and skip quantization? This is a real threat, because it captures multimodality without a codebook. What the codebook adds is a corpus-wide notion of genericness, the idf term, which set-to-set distances lack, along with inspectable codewords. We run MMD and optimal-transport distances on the same sets. If they match the idf-weighted histogram, the finding becomes "distributional comparison, not quantization", and we will report it that way. The idf ablation distinguishes the two explanations.

Why not just conjure images with a diffusion model? Conjuring Semantic Similarity is English-only and costly. It loses verb and adjective semantics, and it struggles on imageless abstractions such as "conscience". Raw prompting of abstract words is poor (Text-to-Image Generation for Abstract Concepts reports 1.86/5 in human ratings). We include generated images as an alternative image source feeding the same histogram pipeline, not as a scoring method.

Proposed Experiments

The primary benchmark is VULIC1000, which mixes concrete and abstract nouns. We stratify it by image dispersion and by external concreteness ratings, reporting P@1, P@5 and MRR. BERGSMA500, which is all concrete, serves as the control where we expect parity with AVGMAX. For graded similarity we use the cross-lingual pairs of Multi-SimLex and English SimLex-999, reporting Spearman $\rho$ per part of speech and concreteness bin. For SimLex-999 the file gives 0.41 for the best models at publication and 0.67 for human agreement. The English-Japanese patent corpus tests whether the codebook can serve as extra nodes in Tamura-style label propagation under the small seed lexicon.

The decisive check is an interaction effect in a regression of reciprocal rank on method, dispersion and their product. The method-by-dispersion term must be positive and significant for histogram versus AVGMAX, and it must shrink toward zero when idf is removed. Secondary checks follow from the claims above. Abstract verbs should show no gain, replicating Hill et al.'s threshold rather than contradicting it. Codewords with the highest weight for translated abstract pairs should be recognizably shared attributes on manual inspection. The advantage should survive the switch from search images to PIN-derived contextual images.