Many image collections can be grouped in several valid ways. Flowers can be grouped by color or by species, and landmarks by country or by architectural style. Recent "customized multiple clustering" methods let a user name the grouping they want, such as "group by species". A language model then turns that request into a list of candidate words, and the image features are aligned to those words. This works well when the words are easy: color names, fruit types. It breaks when the correct groups are long-tail names the language model never lists, or when the user's casual phrasing ("what kind of plant it is") does not match how a knowledge base names the concept (a Wikidata "parent taxon" at genus rank). Any group missing from the word list cannot be recovered later in training.
We propose to build the word list from an external knowledge base instead of from a language model's memory, and to let the image collection decide which knowledge-base values belong on it. The pipeline has three steps. It maps the user's informal facet request to knowledge-base properties. It gathers every value those properties take. It then keeps only the values that the images themselves "claim" under a vision-language model, expanding to taxonomic neighbors of claimed values and repeating. The claim is that the main error in language-guided multiple clustering on fine-grained facets is vocabulary coverage, not representation learning. We also claim that the collection, taken as a whole, gives a feedback signal for closing that gap which no single query has.
Where this sits
The direct predecessor is Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning (Multi-Sub). It asks GPT-4 for reference words for a facet, learns a proxy embedding as a weighted combination of those words in CLIP space, and alternates between aligning images to that proxy subspace and tightening clusters. Its reported results are strong on easy facets: a Rand Index of 0.9964 for color and 1.0000 for species on Fruit. They are weaker on identity for CMUface, where normalized mutual information (NMI, a standard agreement score between a clustering and ground-truth labels) rose from about 0.66 to 0.7441. The file itself names the limitation we target: when the prior model's vocabulary omits categories, "omissions in the vocabulary cannot always be rectified." We keep Multi-Sub's proxy learning unchanged and replace only its source of reference words. That makes the comparison clean.
Earlier multiple-clustering methods motivate the problem but do not supply the mechanism. Multiple Independent Subspace Clusterings, Non-redundant multiple clustering by nonnegative matrix factorization and Deep Embedded Non-Redundant Clustering find alternative partitions without any semantic label. As a result, the user cannot steer them and must interpret their output afterwards.
The idea of retrieving from a knowledge base to fix naming mismatch comes from the knowledge-based visual question answering literature. Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? shows that knowing the correct entity raises a pipeline from 18.2% to 45.6% accuracy on INFOSEEK. In other words, mapping images onto knowledge-base names is the bottleneck. Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering builds entity candidates from tags, captions and CLIP-matched Wikidata entries, but it admits to a fixed descriptor vocabulary. Both work one query at a time. We instead exploit the fact that clustering sees the whole collection, so a value counts as present only if many images support it.
The feedback loop borrows its shape from text retrieval. Synergistic Interplay between Search and Large Language Models for Information Retrieval alternates language-model expansion with retrieval, and RaFe: Ranking Feedback Improves Query Rewriting for RAG trains rewriters from reranker scores. Our feedback signal is the image collection's assignment structure, not a reranker. This avoids inheriting a reranker's blind spots, although it inherits CLIP's blind spots instead, which we test below. InteR also reports drift beyond two iterations, which constrains us to a small, capped number of expansion rounds.
Core mechanism
Given a facet request $f$, a text retriever over Wikidata property and class descriptions returns candidate properties. A small language model rewrites $f$ into knowledge-base phrasing before retrieval. Because the benchmark facets are fixed, this step can be checked against an oracle mapping.
The values of the selected properties form a large candidate set $V$, which can contain tens of thousands of species. For each value $v$, we compute a collection-level support score
$$s(v) = \sum_{i=1}^{N} \mathbb{1}\left[v = \arg\max_{u \in V} \cos(x_i, t_u)\right]\cdot \left(\cos(x_i, t_v) - \max_{u \neq v}\cos(x_i, t_u)\right),$$
where $x_i$ is the CLIP embedding of image $i$, $N$ is the number of images, and $t_u$ is the CLIP text embedding of value $u$. In words, a value earns support from the images for which it is the top match, weighted by how clearly it wins.
We keep values whose support exceeds a threshold. We then add their knowledge-base neighbors (siblings in the taxonomy, or other values of the same class) and rescore, for at most two rounds. The surviving set becomes Multi-Sub's reference words.
When the user requests several facets, a value that is strongly supported under two facets is assigned to the facet whose property it came from. This keeps the vocabularies non-redundant, in the spirit of the multiple-clustering literature.
Research program
The program has three parts. First, we build the method on top of a reimplementation of Multi-Sub.
Second, we build a benchmark in which facets are defined by the knowledge base rather than by hand. The images come from the iNaturalist and Google Landmarks portions of Encyclopedic VQA, whose entities are linked to Wikipedia. From Wikidata we derive facets of increasing difficulty. For species images these are family, genus and species. For landmarks they are instance-of (bridge, cathedral), architectural style and country. This yields multi-facet ground truth at a long-tail scale that the existing Fruit, Cards and CMUface datasets do not reach.
Third, we measure vocabulary recall directly: the fraction of ground-truth groups whose name appears in the reference set. This lets us separate coverage errors from alignment errors for every method.
Baselines
The naive reference point is k-means on frozen CLIP image features, with k set to the true number of groups for each facet. This shows how much any language guidance adds.
The strongest existing method is Multi-Sub with GPT-4 reference words, run as the paper describes. We run it with the same CLIP backbone, the same number of proxy-learning epochs and the same hyperparameter search budget as our variant. Only the reference-word source changes. Multi-Sub's reported numbers on its own datasets are the ones quoted above. The file lists no reported result for Multi-Sub on Flowers, Stanford Cars, Card or CIFAR-10, and none on any Encyclopedic VQA-derived facet, so we rerun it everywhere. Beating it means a higher NMI on genus, species, architectural-style and country facets, averaged over five seeds with non-overlapping confidence intervals, together with no loss on the saturated Fruit facets.
Zero-shot CLIP labeling with our knowledge-base vocabulary, with no proxy learning, tests whether better words alone are enough. Multi-Sub already beat zero-shot CLIP with GPT-4 words, so if our vocabulary closes that gap without learning, the learned subspace matters less than claimed.
Three ablations isolate the key ingredient. The first uses the knowledge-base vocabulary without collection pruning, which asks whether unpruned tens-of-thousands-word lists hurt proxy learning. The second applies collection pruning to a GPT-4 list, which asks whether pruning alone helps. The third replaces the learned facet-to-property mapping with the oracle mapping, which measures how much of the remaining error is the lexical gap between the user's phrasing and the knowledge base.
Why not just…?
Why not just ask GPT-4 for a much longer word list? This is cheap, and on familiar facets it may match us. Two things suggest it will not be enough in the long tail. The file reports that closed-book PaLI-X reaches only 10.8% on INFOSEEK-Human, which indicates that large models' parametric knowledge of tail entities is unreliable. And a longer list without pruning adds distractors. We run GPT-4 at list lengths of 50, 500 and 5000, with and without pruning, and report vocabulary recall. If a pruned long GPT-4 list matches the knowledge-base vocabulary on species and style facets, the knowledge base is unnecessary and we will say so.
Why not just caption every image with a multimodal model and harvest facet words from the captions? This is the most serious threat, because the words are grounded in the data like ours. Captioners, however, tend to name coarse categories ("a purple flower") rather than knowledge-base entities. Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering documents this "plush toy versus Teddy Bear" vagueness. We include caption-harvested vocabulary, with the same pruning, as a full baseline. We expect it to compete on instance-of and color facets and to fall short on genus, species and style. If it does not fall short, the claim fails.
Why not just fine-tune CLIP on the knowledge-base labels? That is supervised classification, not clustering. It needs labeled images for every facet a user might request, whereas our method needs only the facet phrase and the unlabeled collection. We report it as an upper bound, not as a competitor.
Proposed Experiments
The main table gives NMI and Rand Index per facet on the Encyclopedic VQA-derived benchmark and on Multi-Sub's datasets, together with vocabulary recall for every method.
Four decisive checks follow. The first is whether our gain over Multi-Sub grows as facets get finer, from family to genus to species. That trend is the signature of a coverage problem. The second is whether vocabulary recall explains most of the NMI differences across methods; we test this with a regression over facet-method pairs. The third is a negative control on CMUface identity, where no knowledge base names the people. There we expect no gain, and a gain would point to leakage. The fourth probes the pruning signal's dependence on CLIP. We slice results by CLIP's zero-shot accuracy on each facet and ask whether pruning discards true values that CLIP cannot see.
As a transfer probe back to the entity-linking bottleneck, we use the pruned species vocabulary built from the Encyclopedic VQA test images as the candidate entity set for image-to-entity linking. We then measure top-1 entity recall against plain CLIP matching over all 2M entities. For reference, EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering reports 46.1 answer accuracy on Encyclopedic VQA and a 1.3% top-1 recall gain for its entity binding, but the file gives no absolute top-1 figure. This probe is exploratory. A positive result would suggest that looking at a whole collection shrinks the naming gap for single-query retrieval too, but the main claim does not depend on it.
References
- Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning
- Multiple Independent Subspace Clusterings
- Non-redundant multiple clustering by nonnegative matrix factorization
- Deep Embedded Non-Redundant Clustering
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering
- Synergistic Interplay between Search and Large Language Models for Information Retrieval
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering