Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards

Andrey KuznetsovarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces a novel approach to align Visual Language Models (VLMs) with human visual preferences by enabling them to reason about these preferences at test time using reinforcement learning and soft rewards. This method achieves competitive accuracy on preference prediction tasks, offering a more interpretable and generalizable alternative to traditional fine-tuning.

What Problem Does This Paper Address?

Generative AI models, especially those for images and videos, have seen significant improvements when aligned with human preferences. However, collecting this preference data is incredibly resource-intensive. Existing methods often rely on supervised fine-tuning or encoder-based models, which can struggle with generalization when data is scarce. This paper tackles the challenge of achieving robust human preference alignment in VLMs, particularly in low-data regimes, by exploring test-time reasoning as a more efficient and interpretable solution.

Methods

The core of this work lies in enabling VLMs to perform "test-time reasoning" for visual preferences. They achieve this by:

  • Reinforcement Learning (RL) with Group Relative Policy Optimization (GRPO): Inspired by recent LLM advancements, the authors adapt GRPO, a variant of Proximal Policy Optimization. GRPO is designed for efficient RL training by focusing on relative reward comparisons within groups of actions, reducing computational overhead.
  • Soft Rewards: Instead of hard labels or scores, the model learns to predict continuous "soft" rewards. This is done in two ways:
  • Single Image Rating: Predicting a probability distribution over a set of rating tokens (e.g., numerical scores) for an image.
  • Anchor-Based Pairwise Comparison: For a given prompt, an anchor image is randomly selected and compared against other images. The VLM generates a "thinking" trace to reason about the preference, outputting a binary probability of preference.
  • Test-Time Reasoning Strategy: At inference, the VLM uses its reasoning capabilities to process the prompt and images, generating soft scores or probabilities that reflect human preferences. This allows the model to adapt its reasoning process dynamically.

Datasets

The paper utilizes two established datasets for training and evaluation:

  • ImageReward: Contains 137,000 expert-annotated image pairs generated from the same text prompts, where annotators selected the preferred image.
  • Human Preference Dataset v2 (HPSv2): A larger dataset with 798,090 preference choices across 430,060 image pairs, also linked to text prompts, focusing on aesthetic judgment. For their experiments, they used a 25% subset of HPSv2.

Key Insights and Ideas

  • VLMs Can Reason About Human Preferences: The study demonstrates that current VLMs, when trained with RL and test-time reasoning, can effectively capture and predict human visual preferences, going beyond simple pattern matching.
  • Test-Time Reasoning Enhances Generalization: Enabling reasoning at inference time proves more effective for generalization, especially when training data is limited, compared to traditional supervised fine-tuning which can lead to memorization.
  • Soft Rewards Offer Interpretability: The use of soft rewards and reasoning traces provides a more transparent and interpretable output, allowing for a deeper understanding of why a model prefers one image over another.
  • Efficiency in Low-Data Regimes: The approach shows strong performance even when trained on a fraction of the available data (e.g., 25% of HPSv2), highlighting its efficiency and adaptability.

Key Results

  • Competitive Accuracy: The proposed method achieved accuracies of 64.9% on the ImageReward test set and 65.4% on the HPSv2 test set (using a 25% subset for training). These results are comparable to traditional encoder-based models.
  • Improved Performance with Soft Rewards: Compared to zero-shot reasoning, incorporating soft rewards significantly boosted performance on the ImageReward test set, reaching 59.2% accuracy.
  • Human-Level Performance: The soft-reward-enhanced reasoning consistently elevated performance, nearing human-level accuracy.
  • Averaging Reasoning Rollouts: Similar to text-only models, averaging scores from multiple reasoning rollouts further improved accuracy.

Limitations and Future Work

  • Computational Cost of RL: While GRPO is efficient, RL training can still be computationally intensive compared to supervised methods.
  • Further Exploration of Soft Reward Strategies: The authors suggest that exploring more sophisticated soft reward functions and reasoning strategies could lead to further improvements.
  • Broader Applications: The success in visual preference modeling opens doors for applying this test-time reasoning approach to other complex tasks in multimodal AI.
  • Scalability with Larger Models: The paper hints at the potential for even greater gains with larger and more capable VLMs.

Summary updated

Abstract

Can Visual Language Models (VLMs) effectively capture human visual preferences? This work addresses this question by training VLMs to think about preferences at test time, employing reinforcement learning methods inspired by DeepSeek R1 and OpenAI O1. Using datasets such as ImageReward and Human Preference Score v2 (HPSv2), our models achieve accuracies of 64.9% on the ImageReward test set (trained on ImageReward official split) and 65.4% on HPSv2 (trained on approximately 25% of its data). These results match traditional encoder-based models while providing transparent reasoning and enhanced generalization. This approach allows to use not only rich VLM world knowledge, but also its potential to think, yielding interpretable outcomes that help decision-making processes. By demonstrating that human visual preferences reasonable by current VLMs, we introduce efficient soft-reward strategies for image ranking, outperforming simplistic selection or scoring methods. This reasoning capability enables VLMs to rank arbitrary images-regardless of aspect ratio or complexity-thereby potentially amplifying the effectiveness of visual Preference Optimization. By reducing the need for extensive markup while improving reward generalization and explainability, our findings can be a strong mile-stone that will enhance text-to-vision models even further.
BibTeX
@misc{kuznetsov2025test,
  author = {Andrey Kuznetsov},
  title = {{Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards}},
  year = {2025},
  eprint = {2503.19948},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.19948},
}
Figure