TL;DR
This paper introduces VERITAS, a novel pipeline that significantly enhances the quality of multimodal supervised fine-tuning (SFT) data by integrating visual priors and multiple expert Large Multimodal Models (LMMs). VERITAS addresses the critical issue of factual errors and hallucinations in current SFT data, leading to improved downstream model performance, especially in complex reasoning tasks.
What Problem Does This Paper Address?
The performance of Large Multimodal Models (LMMs) is heavily reliant on the quality of their supervised fine-tuning (SFT) data. However, existing methods for generating or enhancing this data often suffer from factual inaccuracies and visual hallucinations. This is largely due to the limitations of current LMMs in accurately perceiving and interpreting visual information, leading to a propagation of errors. Simply filtering out low-quality data can reduce data diversity, while relying on a single LMM for critique introduces individual biases. The challenge is to create high-quality, diverse, and reliable SFT data that grounds LMMs in visual evidence and leverages diverse expert opinions without incurring prohibitive costs.
Methods
VERITAS employs a four-stage pipeline:
- Vision-Prior Extraction: This stage anchors the critique process by extracting structured visual information from images. It uses specialized models like RAM++ for object tagging and PP-OCRv4 for Optical Character Recognition (OCR) to generate textual representations of visual content. These "vision priors" are then combined with the original image, question, and answer.
- Tri-Expert Assessment with Shrinkage Fusion: Three state-of-the-art LMMs (GPT-4o, Gemini-2.5-Pro, and Doubao-1.5-pro) independently critique the answers, providing rationales and scores. A novel domain-aware James-Stein shrinkage method is then used to statistically fuse these individual scores into a single, high-confidence "gold score." This fusion technique balances bias and variance, reducing the impact of individual expert limitations and domain-specific noise.
- Integration with GRPO Critic: To make the evaluation process efficient, the costly ensemble of three LMMs is distilled into a lightweight, 7B-parameter multimodal critic. This critic is trained using Group Relative Policy Optimization (GRPO), which allows it to learn fine-grained ranking differences and achieve GPT-4o-level ranking fidelity at a fraction of the computational cost.
- Self-Refinement and Answer Selection: The LMM experts rewrite the original answers based on the critiques and vision priors. The GRPO critic then selects the best candidate answer from the original and the rewritten versions, yielding a final, confidence-graded, and denoised dataset entry.
Datasets
- Training Corpora: The VERITAS Instruction Corpus, comprising 95,955 samples, is formed by combining seven public multimodal instruction sources. This corpus consists of both original (Raw) and VERITAS-refined (Refine) answer pairs.
- Critic-Evaluation Sets:
- In-domain 1K: 1,000 image-question-answer triplets sampled from the same seven sources as the training data, annotated by human annotators for quality scoring.
- Out-of-domain CLEVR-500: 500 images from the CLEVR test split with automatically injected errors to create three tiers of answer quality (High, Medium, Low).
- Down-Stream Benchmarks: Six widely-used benchmarks were used for evaluation: MME, OCR-VQA, MM-Vet, MathVista, MMT-bench, and POPE.
Key Insights and Ideas
- Vision Priors are Crucial for Grounding: Explicitly incorporating visual information (object tags, OCR text) from specialized models significantly improves LMMs' ability to ground their critiques and evaluations in observable evidence, especially for perception-heavy tasks.
- No Single LMM is an Oracle: Relying on multiple diverse LMM critics, rather than a single one, is essential to mitigate individual biases and capture a broader range of reasoning capabilities.
- Statistical Fusion Outperforms Simple Averaging: Domain-aware shrinkage fusion, which considers domain-specific reliability and noise, leads to more robust and accurate aggregated scores compared to simple averaging.
- Efficient Critics are Achievable: The GRPO-trained lightweight critic demonstrates that high-quality evaluation capabilities can be achieved at a significantly reduced computational cost, making robust data curation more scalable.
- Refinement is More Effective Than Filtering: Generating multiple refined answers and selecting the best one using a critic is superior to simply filtering out low-quality data, as it preserves data diversity while improving quality.
Key Results
- Models fine-tuned with VERITAS-processed data achieved an average accuracy improvement of +7.4 across six benchmarks compared to models trained on raw data.
- VERITAS showed particularly strong gains on perception-centric tasks like OCR-VQA (+14.35) and MME (+14.2).
- The lightweight GRPO critic achieved a Kendall's τ of 0.71 with human judgments, only 0.05 behind GPT-4o, while being two orders of magnitude cheaper.
- The GRPO critic demonstrated robust generalization to out-of-domain data, maintaining a reasonable correlation with human scores.
- VERITAS maintained a hallucination rate comparable to the baseline on the POPE benchmark, indicating that refinement does not introduce new hallucinations.
Limitations and Future Work
- Prompt Length: The critique and rewrite prompts used are relatively long, which increases computational overhead. Future work could explore prompt optimization to maintain performance while reducing costs.
- Dependency on Powerful LMMs: VERITAS relies on state-of-the-art LMMs (GPT-4o, Gemini-2.5-Pro, Doubao-1.5-pro) for critiques and refinements. Access to these models may be restricted due to licensing or API limitations, potentially hindering broader adoption.
- Open-Source Expert Performance: While using open-source LMMs for critiques still yields gains, the performance ceiling is constrained by the strength of these models compared to proprietary ones, especially for reasoning-heavy tasks. Future work could explore further improvements with stronger open-source critics.
Summary updated