TL;DR
While Multimodal Large Language Models (MLLMs) excel at recognizing what is in an image, they struggle to evaluate how an image looks. This paper introduces UniPercept-Bench, a unified benchmark evaluating image aesthetics, quality, structure, and texture, alongside UniPercept, a baseline model trained with task-aligned reinforcement learning that sets new standards in fine-grained perceptual understanding and serves as an effective reward model for text-to-image generation.
What Problem Does This Paper Address?
Current MLLMs are remarkably good at semantic-level tasks like image captioning and object visual reasoning. However, human visual perception involves more than just identifying objects; it requires nuanced judgments about aesthetic harmony, degradation severity, structural regularity, and surface texture.
Existing image assessment tools are highly fragmented, usually focusing on a single dimension (like numerical quality scoring) and failing to provide a holistic, multi-dimensional evaluation framework. MLLMs evaluated on these tasks often produce unstable or hallucinated predictions when forced to output continuous ratings. This paper addresses this gap by unifying these disparate dimensions into a single, comprehensive framework for perceptual-level image understanding.
Methods
The authors tackle unified perceptual understanding through a multi-tiered taxonomy and a two-stage training pipeline:
- Hierarchical Taxonomy: They organize perceptual attributes into three progressive layers: Domain (Aesthetics, Quality, Structure & Texture), Category (e.g., Composition & Design, Distortion Severity), and Criterion (e.g., Visual Balance, Stylistic Classification).
- Domain-Adaptive Pre-Training: To give the model a foundational grasp of low-level visual features, they pre-train an InternVL3-8B base model on a massive corpus of text-based and visual rating (VR) question-answering pairs.
- Task-Aligned Reinforcement Learning: To align the model across both discrete Visual Question Answering (VQA) and continuous Visual Rating (VR) tasks, they employ the GRPO (Group Relative Policy Optimization) algorithm.
- For VQA, they use a standard binary correctness reward.
- For VR, they introduce a novel Adaptive Gaussian Soft Reward. Instead of penalizing any score that doesn't perfectly match the ground truth, this soft reward continuously evaluates predictions based on their numerical deviation, providing smoother gradients and preventing threshold-induced discontinuities.
Datasets
The paper introduces UniPercept-Bench, a new evaluation benchmark containing approximately 6,000 test samples. It spans 3 domains, 17 categories, and 43 criteria, supporting both VR and VQA formats.
For training, the authors compiled a massive dataset from various open-source pipelines:
- Pre-training Corpus: Roughly 800,000 samples aggregated from datasets like APDDv2, Impressions, AVA, TAD66K, Q-Ground-100K, DataDepictQA, SPAQ, KADID, PIPAL, DTD, and FMD.
- RL Training: Utilized specialized visual rating datasets (ArtiMuse-10K, KonIQ-10K, ISTA-10K) combined with ~30,000 generated VQA samples.
Key Insights and Ideas
- Perceptual Synergy: One of the most striking findings is that training a model on all three perceptual domains simultaneously (Aesthetics, Quality, Structure) yields better overall results than training single-domain models. This suggests that these low-level visual tasks share underlying perceptual representations that mutually benefit one another during optimization.
- The "Token As Score" Necessity: General MLLMs struggle to output stable numerical ratings directly in text form. Utilizing structured reinforcement learning strategies stabilizes regression behavior on continuous scales, bridging the gap between quantitative judgment and explanatory consistency.
- Plug-and-Play Generative Guidance: Beyond evaluation, a unified perceptual model can act as a multi-dimensional reward model to fine-tune text-to-image diffusion models (like FLUX.1-dev), allowing direct, controllable improvements in sharpness, clarity, and composition without needing human feedback loops.
Key Results
UniPercept consistently outperformed both massive proprietary models (like GPT-4o and Claude-Sonnet-4.5) and specialized image-assessment models across the board:
- Visual Rating (VR): On the IQA domain (KonIQ-10K), UniPercept achieved an exceptional SRCC/PLCC of 0.940/0.949, outperforming the closest specialized model, Q-Align (0.941/0.940). On the newly defined ISTA-10K structural dataset, it achieved 0.778/0.767, while general models like GPT-4o hovered near zero correlation (-0.003/0.116).
- Visual Question Answering (VQA): UniPercept achieved an overall accuracy of 80.62% across all domains, significantly beating GPT-4o (66.36% on IQA, 74.64% on ISTA) and LLaVA-OneVision-8B.
- Generative Fine-Tuning: When used as a reward model for FLUX.1-dev, the UniPercept-guided model showed clear quantitative gains in image quality (DeQA score rose from 4.32 to 4.33) and aesthetic metrics (ArtiMuse score jumped from 59.02 to 65.52).
Limitations and Future Work
While UniPercept-Bench represents a major step forward for fine-grained visual evaluation, the authors acknowledge that it remains smaller in scale than typical semantic-level benchmarks. Future work will focus on expanding the dataset's scale and exploring deeper integration strategies to further enhance the perceptual capabilities of next-generation MLLMs.
Summary updated