Introduction
In e-commerce, accurate relevance assessment is critical for maintaining user trust and conversion rates. However, existing AI benchmarks for this domain face two significant hurdles: saturation and subjectivity. As smaller models achieve high accuracy on legacy datasets, it becomes increasingly difficult to differentiate between baseline performance and true industrial-grade reasoning. Furthermore, traditional "black-box" labels (e.g., Relevant vs. Irrelevant) lack the standardized logic necessary to leverage and evaluate Chain-of-Thought (CoT) reasoning effectively.
To address these challenges, we introduce RAIR (Rule-Aware benchmark with Image for Relevance), a multimodal benchmark designed to transform relevance assessment into an objective, rule-based protocol.
The RAIR Framework and Relevance Scale
RAIR deconstructs user intent into 16 distinct attribute dimensions, including Category, Brand, Material, and IP. Central to this framework is a fine-grained, four-level relevance scale (L1–L4) that provides a deterministic logic for evaluation:
- L4 (Perfect Match): Ideal alignment between query and item.
- L3 (Partial Match): Semantic proximity with minor attribute conflicts (e.g., Red vs. Burgundy).
- L2 (Explicit Mismatch): Specific, identifiable attribute conflicts (e.g., Blue vs. Burgundy).
- L1 (Completely Inconsistent): Fundamental category or functional errors (e.g., Pants vs. Dress).
Methodology and Dataset Composition
The RAIR benchmark comprises 48,949 samples across 14 industries. The construction pipeline utilizes a hybrid approach, combining regex and LLMs for candidate generation, a tri-model ensemble (Qwen, DeepSeek, Llama) for difficulty verification, and a VLM ensemble to identify visually necessary cases. Professional annotators performed a final audit to ensure a ground truth accuracy of 96.3%.
The dataset is strategically split into three subsets:
- General Subset (32,123 samples): Evaluates fundamental capabilities across balanced industries.
- Hard Subset (10,931 samples): Focuses on long-tail jargon, complex reasoning (e.g., negation), and multi-attribute constraints.
- Visually Salient Subset (5,895 samples): Contains cases where visual evidence is strictly required for accurate judgment.
Key Insights and Experimental Results
Experimental evaluations on RAIR reveal several critical insights into the current state of multimodal LLMs:
- The "Thinking" Advantage: Models designed for extended reasoning (e.g., Qwen-Thinking or o1-style models) significantly outperform standard instruction-tuned models when complex rules are provided. While standard models often struggle with long-context rule adherence, "thinking" models successfully ground their logic in the provided 16-attribute framework.
- Scale and Difficulty Correlation: While performance on the General subset is often high across various model sizes, the Hard Subset restores a clear correlation between parameter scale and performance, making it a superior metric for measuring frontier AI progress.
- Visual Grounding Challenges: Even top-tier Vision-Language Models (VLMs) experience performance degradation on the Visually Salient subset. For instance, while incorporating images improved Gemini 2.5 Pro's performance by 9.2%, smaller models saw negligible gains, suggesting that sophisticated visual grounding is an emergent capability of larger models.
- Benchmark Difficulty: The benchmark remains challenging even for frontier models. GPT-5 achieved only 84.5% binary accuracy and a 43.3% Macro-F1 on the General Subset, indicating substantial room for improvement in industrial relevance tasks.
Conclusion
RAIR shifts e-commerce relevance assessment from subjective labeling to a deterministic science. By providing a rule-aware, multimodal framework with specialized hard and visually salient subsets, RAIR offers a rigorous testing ground for the next generation of AI models. Future work will focus on expanding the benchmark's linguistic reach and using it to improve instruction-following capabilities in complex business logic scenarios.
Summary updated