When you search for a famous landmark, the top results are often a dozen near-identical postcards taken from the same plaza at noon. The standard fix is maximal marginal relevance (MMR), which builds the result list one image at a time. At each step it picks the image whose relevance score, minus a penalty for resembling already-chosen images, is highest. A weight λ sets how much the penalty counts. MMR has a known flaw. The relevance score (often text-to-image) and the resemblance score (image-to-image) live on unrelated scales, yet MMR subtracts one from the other. Once the obvious clusters are used up, the cheapest way to look "different" is to be irrelevant. In visual search this shows up as a steady precision tax. On ImageCLEF 2008, MMR-style diversification raised Cluster Recall@20 from 0.3727 to 0.4146 while Precision@20 fell from 0.5577 to 0.5282 (XRCE's Participation to ImageCLEF 2008).
We propose to replace both MMR scores with calibrated probabilities. The first is the probability that an image is relevant. The second is the probability that two images depict the same facet, meaning the same human-judged visual subtopic such as "night view of the facade" or "interior nave". This second probability is learned from the facet clusters that diversity benchmarks already annotate. With both in probability units, greedy selection maximizes the expected number of distinct relevant facets shown. That quantity rewards an irrelevant image with roughly nothing, however unusual it looks. Our core claim is that this change moves the precision–cluster-recall frontier outward instead of along it. Measured at the same cluster recall, it should keep more precision than tuned MMR, clustering, or determinantal point process baselines. It should also do this with one global setting rather than per-query λ tuning.
Core Claim and Mechanism
Let $r_i$ be the calibrated probability that candidate image $i$ is relevant to the query. Let $p_{ij}$ be the calibrated probability that images $i$ and $j$ belong to the same facet. Given a selected set $S$, we score a candidate by the chance that it is relevant and is not a duplicate facet of any relevant image already chosen:
$$\text{gain}(i \mid S) = r_i \prod_{j \in S} \left(1 - p_{ij}\, r_j\right).$$
Summing these gains gives the expected number of distinct relevant facets in the list, under the simplifying assumption that relevance and same-facet events are independent across pairs. This is a probabilistic coverage objective. It has diminishing returns, so greedy selection carries the usual $(1-1/e)$ approximation guarantee for such functions. Compared with MMR, three things change. The relevance–novelty interaction is multiplicative, so low relevance cannot be offset by high novelty. Redundancy is penalized only to the extent the earlier image was itself likely relevant. And no λ is needed, because both factors are already on the same scale.
The model behind $p_{ij}$ is the central new ingredient. It is a small pairwise classifier on top of frozen image embeddings. Inputs are the two embedding vectors, their difference and product, and, when present, metadata gaps such as GPS distance, time difference, and same-uploader, each with a missing-value indicator. It is trained to predict whether two relevant images share a ground-truth facet cluster in the MediaEval diverse social images development data. It is then calibrated with temperature scaling on held-out queries. Relevance $r_i$ comes from a single shared relevance model, also calibrated, used identically by every method we compare.
Two lessons from the literature shape this design. Semantic Diversity versus Visual Diversity in Visual Dictionaries found that low-level visual variety is essentially uncorrelated with semantic variety. Raw embedding distance therefore conflates "slightly different lighting" with "a genuinely different view", and we instead learn which differences humans count as new facets. Second, the "GPS paradox" from UPMC at MediaEval 2013 showed that photographers standing at the same coordinates produce different facets, so visual descriptors beat GPS. Later systems found that visual-only pipelines collapse in precision (CFM@MediaEval 2017; ETH-CVL @ MediaEval 2015). A learned pairwise model can work out how much each signal matters for facet identity, rather than having it fixed by hand.
Relation to Prior Work
The closest prior systems each combine relevance and redundancy, but none learns redundancy as a calibrated same-facet probability.
Three systems apply MMR directly to visual redundancy, all with an uncalibrated similarity and a hand-tuned λ. XRCE's Participation to ImageCLEF 2008 used Fisher-vector distances and documented the precision loss. NLE@MediaEval'17 used Inception-ResNet features and cross-media similarity. SEDO ran MMR for regional diversity and reported trading accuracy for diversity. We keep the greedy, cheap, one-image-at-a-time character these systems share, but change what is subtracted and how.
ETH-CVL @ MediaEval 2015 is the nearest learned approach. It learns weights over five fixed submodular terms, such as k-medoid representativeness and text relevance, using a structured SVM. It learns how to mix hand-designed terms, whereas we learn the redundancy function itself from facet labels.
MS-DPPs attacks the scale-mismatch problem through tangent normalization of several similarity matrices on a manifold. It depends on clean metadata and costly matrix operations. Our scales are aligned by calibration instead, and our model degrades gracefully when metadata is missing.
Pix2Key runs MMR over interpretable attribute dictionaries for fashion. It explicitly lists camera viewpoint and scene layout, the very redundancy that dominates landmark search, as open. We address that regime without requiring an attribute vocabulary.
Finally, NLE@MediaEval'17 also constrains our design. There, end-to-end joint embeddings failed to beat simple unsupervised neighbor voting, plausibly for lack of task data. We therefore keep the backbone frozen and learn only a small pairwise head, which the modest number of facet-annotated pairs can support.
Research Program
The first step is training and calibrating the same-facet model on MediaEval development queries. We sample positive pairs from shared ground-truth clusters and negative pairs from different clusters within the same query. We test two frozen backbones: an older CNN to match prior systems, and a modern vision-language embedding. We measure pairwise calibration error and AUC on held-out queries.
The second step builds the calibrated greedy reranker. We also build two intermediate variants that separate the contributions. One uses the multiplicative objective with raw cosine similarity in place of $p_{ij}$. The other uses subtractive MMR with calibrated $p_{ij}$.
The third step is analysis. We examine where gains concentrate: queries with many versus few facets, and images with versus without metadata. We also check whether the learned model uses GPS when it is present, as the GPS-paradox result predicts it should only weakly.
The main evaluation trains on MediaEval 2015 data and tests on MediaEval 2017. This cross-year split tests whether the facet notion transfers across query sets, rather than splitting queries within a single year. MediaEval 2013 serves as a secondary test with its @10 metrics, and ImageCLEF 2008 as a check on older photo collections.
Baselines
Every baseline receives the same candidate pool and the same calibrated relevance scores $r_i$. Only the selection step differs. Each gets the same tuning budget: a grid search of comparable size on development queries, with no tuning on test queries.
The naive reference is the relevance-only ranking from our shared relevance model, together with the original Flickr order. For the Flickr order, CFM@MediaEval 2017 reports F1@20 of 0.4277 on the development set. Relevance-only ranking gives the precision ceiling that diversification is supposed to protect.
The most direct competitor is standard MMR as used in NLE@MediaEval'17 and XRCE. We run it on the same modern embeddings, with λ tuned globally on development data. We also run an oracle with λ tuned per test query. A single-setting method that matches this oracle would show the λ problem is solved.
The second competitor is visual hierarchical clustering with round-robin selection, following UPMC at MediaEval 2013 and CFM@MediaEval 2017. We tune the cutoff distance rather than fixing 50 clusters, which makes this baseline stronger than the originals.
The third competitor is a learned submodular mixture in the style of ETH-CVL. We also run a greedy determinantal point process whose kernel factors into relevance-weighted similarity. Compared with standard MMR, this is the strongest structurally multiplicative competitor, and it uses the same embeddings.
The strongest reported number on our main test set is the text-plus-credibility system of CFM@MediaEval 2017: Precision@20 of 0.6881 and F1@20 of 0.6533 on the official test set. For the MMR-based submission on that benchmark, NLE@MediaEval'17, the file contains no scalar results, so we report "no reported result found on Lacuna for NLE@MediaEval'17 P/CR/F1@20" and will re-run it. On MediaEval 2013, the reference is CR@10 of 0.426 and F1@10 of 0.547 from UPMC, whose text-only run reached P@10 of 0.844.
Beating the field means two things on MediaEval 2017. First, F1@20 must exceed both our re-run tuned MMR and 0.6533. Second, at matched CR@20, Precision@20 must be significantly higher than tuned MMR and the determinantal baseline, under paired bootstrap over queries.
The two variants from the research program serve as ablations, isolating calibration from the multiplicative form. If either variant alone recovers most of the gain, the claim narrows to that ingredient.
Why not just…?
Why not just use modern embeddings with plain MMR and a well-tuned λ? This is the most serious threat. Better features may shrink the difference between "different lighting" and "different facet". Penatti et al.'s flat relationship between visual and semantic diversity (Semantic Diversity versus Visual Diversity in Visual Dictionaries) gives reason to doubt that any unsupervised distance captures human facet boundaries. However, that evidence comes from older dictionary-based features. Our comparison against globally tuned and per-query oracle MMR on identical embeddings settles the question. If global MMR matches us, the contribution reduces to "use better embeddings", and we will report that.
Why not just cluster and round-robin with an adaptive cluster count? Clustering makes hard partitions and still treats distance as facet identity. An image on the border of two clusters is either fully redundant or fully novel. A tuned-cutoff clustering baseline is included. If it matches us, the learned pairwise model, not the soft objective, is doing the work. The ablation that feeds learned $p_{ij}$ into clustering distinguishes these.
Why not just use a determinantal point process, which already multiplies quality by diversity? A quality-weighted determinantal kernel shares our multiplicative intuition. MS-DPPs reports outperforming MMR, reaching a harmonic mean of 0.8649 on the Visual Genome time task. Its similarity is still raw, though, and the method needs metadata. We test the determinantal baseline with the same embeddings, and also with learned $p_{ij}$ inserted as its kernel. If the latter ties with ours, the contribution is the learned facet probability, and the choice of objective is secondary.
Why not just rerank by text and metadata? Text-and-credibility reranking is the strongest reported system on MediaEval 2017. Identical tags, however, cannot tell two identical postcards apart, and metadata is often missing: 60% of images lacked GPS in MediaEval 2013. We also report results on the subset of images without metadata, where text-only methods should fall behind.
Decisive Checks
The idea succeeds if three things hold under a single global setting. Its precision–cluster-recall curve on MediaEval 2017 must dominate tuned MMR and the determinantal baseline. It must match or beat per-query oracle MMR. And its gain must survive the cross-year transfer from 2015 to 2017.
The idea fails if raw cosine similarity in the multiplicative objective does as well as learned $p_{ij}$. That would mean calibration adds nothing. It also fails if the gains vanish with modern embeddings, which would mean the precision tax was a symptom of weak features rather than of MMR's arithmetic. We will additionally report α-nDCG@20 and ERR-IA@20 from the MediaEval protocol, and calibration curves for $p_{ij}$. Large miscalibration on test queries would undermine the method's justification even if the ranking metrics improved.
References
- XRCE's Participation to ImageCLEF 2008
- Semantic Diversity versus Visual Diversity in Visual Dictionaries
- UPMC at MediaEval 2013: Relevance by Text and Diversity by Visual Clustering
- CFM@MediaEval 2017 Retrieving Diverse Social Images Task via Re-ranking and Hierarchical Clustering
- ETH-CVL @ MediaEval 2015: Learning Objective Functions for Improved Image Retrieval
- NLE@MediaEval'17: Combining Cross-Media Similarity and Embeddings for Retrieving Diverse Social Images
- Enhancing Vision-Language Models for Global Cultural Understanding through Semantic Expansion and Diversity Reranking
- MS-DPPs: Multi-Source Determinantal Point Processes for Contextual Diversity Refinement of Composite Attributes in Text to Image Retrieval
- Pix2Key: Controllable Open-Vocabulary Retrieval with Semantic Decomposition and Self-Supervised Visual Dictionary Learning