Abstract
We present DRESS, a large vision language model (LVLM) that innovatively exploits Natural Language feedback (NLF) from Large Language Models to enhance its alignment and interactions by addressing two key limitations in the state-of-the-art LVLMs. First, prior LVLMs generally rely only on the instruction finetuning stage to enhance alignment with human preferences. Without incorporating extra feedback, they are still prone to generate unhelpful, hallucinated, or harmful responses. Second, while the visual instruction tuning data is generally structured in a multi-turn dialogue format, the connections and dependencies among consecutive conversational turns are weak. This reduces the capacity for effective multi-turn interactions. To tackle these, we propose a novel categorization of the NLF into two key types: critique and refinement. The critique NLF identifies the strengths and weaknesses of the responses and is used to align the LVLMs with human preferences. The refinement NLF offers concrete suggestions for improvement and is adopted to improve the interaction ability of the LVLMs—which focuses on LVLMs' ability to refine responses by incorporating feedback in multi-turn interactions. To address the non-differentiable nature of NLF, we generalize conditional reinforcement learning for training. Our experimental results demonstrate that DRESS can generate more helpful (9.76%), honest (11.52%), and harmless (21.03%) responses, and more effectively learn from feedback during multi-turn interactions compared to SOTA LVMLs.
Introduction
While Large Vision-Language Models (LVLMs) like LLaVA have become impressive at describing images and following instructions, they suffer from two major alignment gaps. First, they frequently generate responses that are unhelpful, hallucinated (dishonest), or potentially harmful—a set of issues known as the "3H" deficiency. Standard Supervised Fine-Tuning (SFT) often fails to provide the nuanced feedback needed to correct these behaviors. Second, most multi-turn datasets are essentially a series of independent questions, meaning models struggle to actually interact and refine their previous answers based on user feedback.
Methodology: Natural Language Feedback
The core innovation of DRESS is the categorization and utilization of Natural Language Feedback (NLF), generated by a "teacher" model (GPT-4). Unlike simple numerical rewards, NLF provides a dense signal that explains why a response is sub-optimal and how to fix it.
- Critique NLF: This identifies the specific strengths and weaknesses of a model's response (e.g., identifying hallucinations). This is used to align the model with human preferences.
- Refinement NLF: This provides concrete suggestions for improvement, teaching the model the meta-skill of iterative improvement during multi-turn interactions.
To address the non-differentiable nature of text-based feedback, the researchers used Generalized Conditional Reinforcement Learning. DRESS is trained to generate responses conditioned on these feedback strings, allowing it to steer toward "excellent" outputs at inference time.
Datasets and Training
The study utilized several key data sources, including the LLaVA Visual Instruction Data and COCO for dense image annotations. A significant contribution is the new NLF Dataset, a collection of 63,000 annotated samples covering the 3H aspects. Additionally, the authors introduced VLSafe, a dataset containing 5,874 samples of adversarial prompts designed to test the model's resistance to generating harmful content.
Key Results
DRESS outperformed state-of-the-art (SOTA) models across all primary metrics:
- Helpfulness: Improved by 9.76% over previous models on open-ended VQA.
- Honesty: Reduced hallucinations by 11.52% (measured by the CHAIR metric).
- Harmlessness: Showed a massive 21.03% improvement in resisting adversarial prompts compared to SFT-only models.
Furthermore, DRESS demonstrated a significantly higher capacity for multi-turn interaction. As shown in the evaluation across four turns, DRESS consistently improves its helpfulness by incorporating feedback, whereas baseline models often plateau or decline in quality as the conversation progresses.
Conclusion
DRESS demonstrates that natural language is a superior reward signal for aligning LVLMs with human values. By training on critiques and refinements, the model not only becomes more helpful, honest, and harmless but also masters the ability to iterate on its own outputs. Future work will explore extending this feedback loop to more complex scenarios like Embodied AI and autonomous web navigation.
Summary updated