TL;DR
Current Multimodal Large Language Models (MLLMs) struggle with visual rating tasks—like assessing image quality, grading medical conditions, or guessing a person's age—because vision encoders are tuned for high-level semantics rather than granular numeric scales. STORM introduces a 655K-sample benchmark spanning 14 datasets and 5 distinct domains, alongside a hierarchical, coarse-to-fine Chain-of-Thought (CoT) reasoning pipeline paired with hybrid text-and-number labels to teach MLLMs unified ordinal regression.
What Problem Does This Paper Address?
Quantifying visual attributes along an ordered scale—known as ordinal regression (OR)—is critical for tasks like Image Quality Assessment (IQA), Image Aesthetic Assessment (IAA), facial age estimation, and clinical disease staging.
Despite their semantic prowess, MLLMs perform poorly on visual rating due to three main roadblocks:
- Label Inconsistencies: Different domains use vastly different rating systems (e.g., discrete Likert-style tiers vs. continuous age numbers), making multi-task training fragile.
- Numeric Hallucination: Vision-language models (typically aligned via CLIP-like objectives) map images to natural language semantics, not continuous numbers, leading to severe hallucination when outputting raw scores.
- Weak Zero-Shot Generalization: Prior rating models are narrowly trained on single tasks with fixed bins, failing to generalize to new scales or unseen domains.
Methods
STORM re-conceptualizes visual rating as a unified, hierarchical ordinal regression task using Qwen2.5-VL-3B as its backbone:
- Dual-Form Labels (Text + Numbers): Instead of forcing the model to predict isolated digits, labels blend semantic descriptions with numeric bounds (e.g.,
Youth (21–30 years old)). This aligns with the pre-trained model's language biases while preserving ordered numeric rank. - Coarse-to-Fine Chain-of-Thought (CoT): The model is prompted to break rating down into intermediate, hierarchical steps:
- Coarse Classification: Partition candidates into broader groups (e.g.,
Below Fair,Fair,Above FairorChild,Teenager,Youth,Adult,Elder). - Fine-Grained Prediction: Pinpoint the exact sub-tier or continuous value within that coarse bracket.
- Self-Consistency Check: Enforce that the fine prediction strictly falls within the selected coarse bucket, pruning improbable candidates and stabilizing training on imbalanced tails.
- Training Regimen: The authors use full fine-tuning (FFT) with FSDP over a two-stage curriculum (pre-training vision-text alignment, followed by visual rating instruction tuning).
[Input Image]
--> Vision Encoder
--> Coarse Category Candidate Prompt
--> Coarse Intermediate Prediction ("Youth")
--> Fine-Grained Refinement ("Youth, 27 years old")
Datasets
The authors compile STORM, comprising 655K question-answer pairs (with a 250K "Lite" version for faster training) across 14 public datasets in 5 domains:
- Image Quality Assessment (IQA): SPAQ, ChallengeDB, KonIQ-10k.
- Image Aesthetics Assessment (IAA): Aesthetics Dataset, AVA, TAD66K.
- Facial Age Estimation (FAE): Adience, CACD, MORPH, UTKFace.
- Medical Disease Grading (MDG): EyePACS, DeepDR, APTOS 2019 (Diabetic Retinopathy).
- Historical Date Estimation (HDE): Palermo et al.'s HCI dataset (1930s–1970s color photos).
Zero-shot generalization was specifically evaluated on held-out test splits from HCI, CACD, UTK, Aesthetic, KonIQ, and APTOS.
Key Insights and Ideas
- Hierarchical candidate pruning eases ordinal learning: Unlike standard classification where classes are independent, adjacent categories in ordinal regression share mathematical proximity. Grouping adjacent labels into coarse buckets simplifies classification complexity and naturally mitigates long-tail class imbalance.
- Language bridges the "number sense" gap: Prompting with descriptive text definitions generated via GPT-4 (and refined manually) significantly outperforms pure numeric prompts. CLIP-based backbones understand semantic descriptors like "mild" or "severe" far better than arbitrary numerical scales.
- Cross-domain transferability of ordinality: Jointly training across all 5 domains yields superior results compared to training domain-specific models, proving that ordinal reasoning itself is a generalizable skill across visual modalities.
- Full Fine-Tuning trumps parameter-efficient methods: Full fine-tuning (FFT) heavily outperformed LoRA (0.533 vs. 0.289 average accuracy), suggesting that adjusting low-level visual perception and ordinal token distributions requires updating base weights.
Key Results
- Overall Benchmark Superiority: STORM-3B achieved an average accuracy of 0.542 (Full) and 0.533 (Lite), with a mean absolute error (MAE) of 1.907 / 1.958, outperforming standard MLLM baselines like zero-shot Qwen2.5-VL-3B (0.288 ACC / 2.358 MAE) and LLaVA-1.5-7B (0.189 ACC / 3.530 MAE).
- Zero-Shot Transfer: On unseen datasets like the Aesthetic dataset, STORM boosted accuracy by nearly 2.5× over vanilla Qwen2.5-VL baselines without CoT. It also achieved zero-shot success on completely unseen temporal estimation (HCI dataset: 0.424 ACC).
- Ablations on Prompting: Coarse-to-fine CoT using hybrid **Num.
- Text reached 0.533 ACC, clearly beating non-CoT (0.466 ACC), Num-only (0.487 ACC), and Text-only (0.485 ACC**) variants.
- Scale Invariance: Scaling from 3B to 7B parameters showed negligible returns (0.533 vs. 0.432 average ACC on the Lite split), making the compact 3B model the ideal tradeoff for efficiency and performance.
Limitations and Future Work
- Prompt Sensitivity: The model remains somewhat sensitive to prompt wording; suboptimal text definitions for rating tiers can introduce variance into model predictions.
- Reliance on LLM Data Generation: Intermediate text prompts and definitions relied on the GPT-4 API, inheriting potential training set biases (partially mitigated through manual post-processing scripts).
- Clinical/Real-World Deployment: The authors note that the model is strictly intended for research benchmarking and is not yet reliable for direct medical diagnostic decision-making. Future work involves extending the framework to video quality assessment and broader scientific regression benchmarks.
Summary updated