Shopping search engines do not rank products by relevance alone. They mix a relevance estimate with business terms such as profit margin, so a pricier item that is nearly as relevant can move up the list. More and more, the relevance part comes from vision-language models, meaning models that read both a product's text and its image. These models have a known weakness. They often answer from the text and do not check the picture. On charts, accuracy reaches 79% when the title gives the answer away and falls to about 30% when the title is removed (Plot Twist). In a catalog, the equivalent failure is a title that says "12-pack, leather" above an image showing a single synthetic item. Our argument is that the revenue term makes this failure much more costly. Profit weighting lifts high-margin items, so a relevance error on such an item jumps to the top of the page instead of staying hidden at rank 40.
The proposal has two parts. First, we measure how much multi-objective ranking amplifies text-shortcut relevance errors. Second, we build a selective verifier that runs slow, deliberate visual checking on only a small set of candidates. It picks the candidates whose position in the business-weighted list would change most if their relevance estimate turned out to be wrong. The ranking objective decides where the perception gap costs money, and the reasoning budget is spent there.
Core Claim
Take the score used by joint profit-relevance rankers in the style of Joint Optimization of Profit and Relevance, $s_i = \hat r_i + \mu\, m_i$. Here $\hat r_i$ is the cheap model's relevance estimate for item $i$, $m_i$ is the item's margin, and $\mu$ sets the trade-off. We make two claims.
First, when text-image conflicts are correlated with margin, the drop in list quality caused by those conflicts grows with $\mu$. The reason is mechanical: a larger $\mu$ pushes conflicted high-margin items into the slots where users actually look. Sellers with expensive items have the strongest incentive to write optimistic titles, so we expect this correlation to exist in real catalogs.
Second, verification should be allocated by the value of fixing an item's relevance estimate, not by model uncertainty alone and not by rank alone. Let $p_i$ be a cheap estimate of the probability that $\hat r_i$ is a shortcut error, obtained from a disagreement score between the title and the image. Let $U(\pi)$ be the list utility of ranking $\pi$, which combines discounted relevance and margin. Then
$$\text{priority}_i = p_i \cdot \lvert U(\pi) - U(\pi^{(i)}) \rvert,$$
where $\pi^{(i)}$ is the list obtained by re-sorting after replacing item $i$'s relevance with its corrected value under the suspected error. Computing $\pi^{(i)}$ requires only a re-sort, so the priority is cheap to evaluate. We verify the top-$B$ items by priority, where $B$ is the budget.
The novel part is this criterion, and its dependence on $\mu$ and $m_i$ is the specific bet. If allocating by uncertainty alone, or by rank alone, does as well at the same budget, the claim is wrong.
Relation to Prior Work
Two lines of work motivate the problem.
Multi-objective ranking assumes relevance inputs are trustworthy. - Louca et al. (Joint Optimization of Profit and Relevance) fold expected revenue into a logistic-regression objective. On Etsy they report AUC 0.6266 and +9.50% profit at the top position. - Bae et al.'s list-wise CPP model (Ranking Items by the Current-Preferences and Profits) reports about +24% profit and about +4% accuracy over the best competitor on Amazon categories. - CLOES (Cascade Ranking for Operational E-commerce Search) adds differentiable penalties for latency and compute.
All three treat items as feature vectors and take relevance as given. We keep their scoring rules and question the relevance input they rely on.
Multimodal models perceive the parts but reason poorly about them, and lean on text. - MATH-Vision (Measuring Multimodal Mathematical Reasoning with MATH-Vision) attributes 42.2% of GPT-4V's errors to reasoning and 31.9% to recognition. - Chart101 (Plot Twist) shows the title shortcut directly.
These papers supply the failure mode we expect to appear in catalog matching.
Other work supplies the verifier's methods. - Perception-time scaling (Unleashing Perception-Time Scaling) turns one-shot visual guesses into deliberate measurement. It raises high-precision accuracy on DisTANCE from 8.0% to 64.7%. - StructChart (StructChart) shows that extracting structured triplets before reasoning beats flat linearization.
We use both ideas. The verifier extracts (attribute, value) pairs from the image alone, compares them with the query's requirements, and only then outputs a corrected relevance.
Two papers constrain the design. - MM-Verify (MM-Verify) needs 12 rollouts to reach 65.3 on MathVista. - RaCT (RaCT) runs at 5.72 seconds per text reranking call. CLOES operates at around 23 ms.
Deliberate reasoning therefore cannot run on every candidate, and this is why the allocation problem matters.
The closest work is SCOUT, then RaCT and Rank2Tell. - SCOUT (SCOUT) combines structured visual chain-of-thought with multiple process rewards. Its rewards are internal to one question-answering task, and it never ranks or weighs business value. - RaCT ranks with chain-of-thought, but only over text and only for relevance. - Rank2Tell (Rank2Tell) ranks visual objects by importance with a single objective.
We deliberately do not train an end-to-end model to "reason about margin." Margin is a known number, and the ranker already handles it exactly.
Research Program
The amplification study. We build a catalog testbed starting from Bae et al.'s Amazon categories (beauty, games, cellphones) and their chronological 60/20/20 splits, using price as the margin proxy. We will need product images for these items. The file does not say whether they ship with the data, so obtaining them is a first feasibility check.
Into a controlled fraction of items we inject title-image conflicts on checkable attributes: pack count, color, material, and included accessories. The injection copies the Chart101 manipulation by editing the title while keeping the image. The correlation between conflict and price is a dial, running from zero to strongly positive. We also mine naturally occurring conflicts by having annotators label a sample of real listings.
We then sweep $\mu$ for both a Louca-style pointwise ranker and CPP. For each setting we measure relevance-weighted ranking quality (nDCG@10 against attribute-correct relevance) and realized profit. Profit counts only items that are truly relevant, since a conflicted item would plausibly be returned. The first claim predicts that the harm from conflicts rises with $\mu$ when conflicts are correlated with price, and stays flat when they are not.
The verifier. We build it on an open vision-language model such as Qwen2.5-VL-7B, the base model used by both SCOUT and the perception-time scaling work. It is a short structured routine that works in three steps:
- Extract attribute triplets from the image with the title hidden.
- Compare them with the attributes the query asks for.
- Emit a corrected relevance value.
We will fine-tune it with GRPO, a reinforcement-learning method that compares groups of sampled answers to one another. Rewards are verifiable: attribute correctness on the injected and annotated items, plus a format reward. We include a title-blind control so the verifier cannot pass by reading the text. The cheap disagreement score $p_i$ comes from a small model that compares title embeddings with image embeddings.
The allocator. The allocator computes the priority above for each query's top-$N$ candidates and verifies the top $B$. We sweep $B$ from 1% to 20% of candidates. At each budget we report the measured added latency next to the CLOES and RaCT reference points.
Baselines
No verification. The reference point is the multi-objective ranker fed unverified cheap relevance, run on the identical testbed. It bounds the harm from below.
Strongest ranking method. We use CPP (Ranking Items by the Current-Preferences and Profits), the list-wise profit-aware ranker. It reports about +24% profit and about +4% accuracy over its best competitor on these same Amazon categories. We rerun it on our splits with its published settings and a tuning budget equal to our own, and we use it as the ranker under every verification policy. That way the comparison isolates the effect of verification. The Louca-style pointwise ranker is run alongside, because its AUC 0.6266 and +9.50% top-1 profit on Etsy are the clearest reported numbers for the joint-loss family. Etsy logs are not public, so we report its behavior on our testbed rather than claiming to reproduce those numbers.
Competing verification policies. All of them use the same verifier, the same ranker, and the same budget $B$:
- uniform verification of the top-$B$ ranked items;
- verification by uncertainty alone ($p_i$ without the utility term), which is the ablation that removes the objective-aware ingredient;
- verification by margin alone;
- random verification;
- verifying every candidate, which serves as the ceiling.
No reported result found on Lacuna for any budgeted verification policy in ranking, so all of these are run fresh.
What counts as success. The proposal wins if, at budgets of 10% or less, the priority policy recovers significantly more of the gap between no verification and full verification than both top-$B$ and uncertainty-only. Utility is measured as attribute-correct nDCG@10 plus realized profit, across seeds and categories. The advantage should grow with $\mu$.
Verifier components. We compare deliberate structured extraction with direct one-shot judgment by the same model, and with a title-visible variant. If the title-visible verifier matches the others on injected conflicts, it is reading text and not images.
Why not just…?
Why not just verify the top-$k$ results? This is the most serious threat. At high $\mu$ the top of the list is already made up mostly of high-margin items, so top-$k$ might approximate our priority closely. The priority criterion should win in two places:
- conflicted items sitting just below the cutoff, which an error moves into view;
- correctly relevant items wrongly pushed down, which top-$k$ never looks at.
If the experiments show top-$k$ within noise of our policy across the $\mu$ sweep, we will report that the objective-aware criterion is unnecessary. The remaining contribution would then be the amplification measurement plus the verifier itself.
Why not just fine-tune a better vision-language relevance model and skip verification? Shortcut reliance does not disappear with ordinary training. The Chart101 models include ChartLlama, which is fine-tuned on charts, and it still reaches only 4.2% on scatter plots. RaCT also documents that specialist ranking fine-tunes can wipe out general ability, with RankZephyr scoring 0.0 on MMLU. We still include a strong fine-tuned relevance model as an extra baseline. If it removes the amplification effect on its own, selective verification has no job.
Why not just blend scores, or train one model with a margin reward the way SCOUT does? Blending does not fix the problem, because it inherits whatever errors the relevance estimate carries. Louca et al. show that naive expected-revenue sorting falls into the price trap. A SCOUT-style margin reward inside the chain-of-thought asks the model to reason about a number the ranker already knows exactly. It also gives the model an incentive to over-rate high-margin items. We include this end-to-end variant and check whether it inflates relevance for expensive conflicted items.
Why not just run MM-Verify-style rollouts on everything? Twelve rollouts per item is incompatible with the tens-of-milliseconds budgets reported by CLOES and MOBIUS (MOBIUS). The full-verification ceiling shows how much would be gained if latency were free. The budget sweep shows how much of that gain is kept at a cost the system can afford.
Decisive Checks
Three results settle the proposal:
- Amplification. Harm from conflicts must grow with $\mu$ under margin-correlated conflicts and not under uncorrelated ones. Without this, the motivation fails.
- Allocation. The priority policy must beat both top-$B$ and uncertainty-only at equal budget. Without this, the method fails, and we would report it as a negative result.
- Real perception. The verifier's gains must survive title removal and transfer to Chart101's title-stripped setting. Otherwise it is exploiting the same text shortcut it was built to catch.
We additionally report the verifier's MathVista accuracy before and after fine-tuning, to check that training for catalog checking does not erode general visual reasoning. We compare against the GPT-4V figure of 49.9% from MathVista only as context, not as a target.
References
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Joint Optimization of Profit and Relevance for Recommendation Systems in E-commerce
- Ranking Items by the Current-Preferences and Profits: A List-wise Learning-to-Rank Approach to Profit Maximization
- Cascade Ranking for Operational E-commerce Search
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Unleashing Perception-Time Scaling to Multimodal Reasoning Models
- StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding
- MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verification
- RaCT: Ranking-aware Chain-of-Thought Optimization for LLMs
- SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward
- Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
- MOBIUS: Towards the Next Generation of Query-Ad Matching in Baidu's Sponsored Search
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts