TL;DR
This paper introduces MarketImage, a diagnostic framework designed to determine if online transaction prices can serve as a low-cost proxy for human visual preference. The authors find that transaction prices are a noisy and often contradictory signal: only about a third of non-tied price directions actually align with human visual rankings, and complex market-inspired elicitation methods (like simulated auctions) often degrade the performance of Multimodal Language Models (MLMs) rather than improving it.
What Problem Does This Paper Address?
Training and evaluating generative vision models currently depends on expensive human annotations or reward models. In contrast, market transaction prices are abundant and reflect "real-world" value. However, a higher price doesn't always mean an image is "better" to the human eye; it might just be rarer, come from a famous artist, or have been sold during a period of high demand.
The authors tackle the identifiability challenge: distinguishing between visual preference signals and non-visual market noise (like provenance or competition). They investigate whether transaction prices can be used as "ground truth" labels for visual preference and whether modern MLMs can recover these signals through various elicitation interfaces.
Methods
The researchers developed the MarketImage framework, which consists of three core stages:
- MarketImageArt: A new dataset pairing real auction prices with price-blind human rankings.
- MarketImageLens: A diagnostic protocol that breaks down image pairs into three categories: price-human agreement, price-human conflict, and price ties. It also audits for "presentation-order effects"—the tendency for humans or models to favor items based on their position in a list.
- MarketValuator: A suite of six elicitation methods used to test MLMs (like GPT-5.5 and Qwen3-VL). Instead of just asking a model "which image is better?", they tested more complex scenarios, such as:
- Budgeted Single-agent: The model must allocate a fixed budget (e.g., €600) across six images.
- Valuator: A multi-agent setup where three model "profiles" bid on items, and the final score is the second-highest bid (simulating a Vickrey auction).
Datasets
The study introduces MarketImageArt, which contains:
- 1,596 artworks from emerging-art auctions (2020–2024).
- 266 six-image sets partitioned into low, mid, and high-price strata.
- 3,990 within-set image pairs with both realized transaction prices and independent, price-blind human rankings.
The prices in the dataset range from €10 to €395, focusing on a "near-tie" regime where visual differences should, in theory, be the primary driver of price.
Key Insights and Ideas
- Price is a Sparse Signal: In the dataset, 57.9% of sets had all six prices tied. Only 24.2% of all image pairs had a clear price difference.
- The Price-Human Gap: Even when a price difference existed, it only agreed with the human majority 34.7% of the time. In many cases, the market and the human eye were in direct conflict.
- Order Bias is Real: The study found a significant "earlier-item" bias. Humans favored the earlier-positioned item in 65.9% of pairs. Because the dataset was sorted by price (lowest to highest), this created a massive confound where the "market" favored the later items while humans favored the earlier ones.
- Mechanism Harm: One might assume that making a model act like a strategic bidder would improve its valuation accuracy. Surprisingly, for the Qwen model family, adding budget constraints and multi-agent "auctions" significantly lowered performance compared to simple direct scoring.
Key Results
- Recoverable Accuracy: On the subset where prices and humans actually agreed (the "protocol-supported" set), HPSv2 (a static visual preference model) performed best at 66.4% accuracy.
- MLM Performance: GPT-5.6 Sol Pair reached 62.1%, while Qwen3-VL-4B Pair struggled significantly at only 8.4%, largely due to its extreme sensitivity to the order of images.
- Elicitation Effects: For GPT models, complex interfaces (like the Valuator) showed slight point gains, but these were not statistically significant. For Qwen, the Valuator interface was "the most harmful tested contrast," dropping accuracy by 26.6 points compared to a single-agent baseline.
Limitations and Future Work
The authors acknowledge several limitations:
- Scope: The data comes from a single, low-price emerging art source. Results might differ in high-end markets (e.g., Sotheby's) where provenance and investment value dominate even more.
- Artificial Sets: The six-image sets were constructed for the experiment, not taken from simultaneous real-world auctions.
- Order Confound: While they audited for order bias, the fixed-order nature of the human annotations remains a significant hurdle for establishing a clean "visual-only" signal.
Future Work: The authors suggest testing broader markets with randomized presentation orders to further untangle the complex relationship between what we see and what we are willing to pay. Not discussed in the paper is the potential for training models specifically to ignore market noise, which this diagnostic framework could facilitate.
Summary updated