VERO: Verification and Zero-Shot Feedback Acquisition for Few-Shot Multimodal Aspect-Level Sentiment Classification

Kai Sun, Hao Wu, Bin Shi, Samuel Mensah, Peng Liu, Bo DongAAAI 2025 · DOI

TL;DR

This paper introduces VERO, a novel acquisition function for few-shot multimodal aspect-level sentiment classification (MALSC) that leverages the unique capabilities of Large Vision-Language Models (LVLMs). VERO intelligently selects challenging samples by analyzing the divergence between an LVLM's initial zero-shot predictions and its subsequent self-verification, leading to significant performance improvements over existing methods.

What Problem Does This Paper Address?

Multimodal Aspect-Level Sentiment Classification (MALSC) aims to determine the sentiment polarity of a specific aspect within a text-image pair. Traditional approaches often require large amounts of labeled data, which is expensive and time-consuming to collect. While few-shot learning offers a solution by using a small number of labeled examples, effectively selecting these informative samples for training is crucial. Existing methods for sample selection often rely on general strategies that don't fully exploit the advanced reasoning and self-correction abilities of modern Large Vision-Language Models (LVLMs). This paper addresses the challenge of efficiently fine-tuning LVLMs for few-shot MALSC by developing a smarter way to pick the most beneficial training examples.

Methods

The core of VERO lies in its acquisition function, which intelligently selects samples for fine-tuning LVLMs. The process involves two key stages:

  1. Zero-Shot Prediction and Self-Verification: For each unlabeled sample, the LVLM first makes a zero-shot prediction of the sentiment polarity. Then, it performs a self-verification step, essentially re-evaluating its initial prediction.
  2. Divergence-Based Sample Selection: VERO identifies samples where the zero-shot prediction and the self-verification prediction diverge. These "changed" samples are considered challenging because they indicate areas where the model is uncertain or makes mistakes. Specifically, VERO focuses on samples within a low-uncertainty region of these "changed" classifications, meaning the model is confident in its new prediction after self-verification. This confidence in a changed prediction signals a particularly informative sample for fine-tuning.

To ensure stable training and prevent overfitting to only difficult examples, VERO also incorporates a small set of "easy" samples (those where zero-shot and self-verification are consistent) to balance the training data.

Datasets

The paper evaluates VERO on two benchmark datasets for MALSC:

  • Twitter-2015
  • Twitter-2017

These datasets are commonly used in MALSC research and were used in their preprocessed forms from prior work to ensure fair comparison. The experiments were conducted in 1% and 7% few-shot settings.

Key Insights and Ideas

  • LVLMs' Self-Verification is a Powerful Signal: The paper strongly suggests that the self-verification mechanism in LVLMs is not just for error correction but is a rich source of information for identifying challenging samples. The divergence between initial and verified predictions is more informative than relying solely on zero-shot predictions or general uncertainty measures.
  • Focusing on "Confident Changes" is Key: VERO's success hinges on selecting samples where the LVLM confidently changes its mind after self-verification. This indicates a sample that truly challenges the model's understanding.
  • Balancing Challenging and Easy Samples is Crucial: While challenging samples are vital, including a small proportion of "easy" samples (where the model is consistent) helps maintain stable performance and prevents overfitting, leading to better generalization.
  • LVLMs Need Fine-Tuning for Specific Tasks: Even powerful LVLMs like LLaVA benefit significantly from task-specific fine-tuning, especially in few-shot scenarios. VERO's acquisition strategy makes this fine-tuning process much more effective.

Key Results

VERO demonstrated superior performance on the Twitter-2015 and Twitter-2017 datasets in few-shot MALSC settings.

  • It achieved an F1 score improvement of up to 6.07% compared to previous methods.
  • When fine-tuning LLaVA-7b and LLaVA-13b models, VERO showed significant gains (at least 8% accuracy and 11% weighted-F1) over the vanilla LVLMs.
  • VERO also outperformed state-of-the-art methods like MultiPoint, achieving comparable or better performance.
  • Ablation studies confirmed that the uncertainty-based selection and the inclusion of self-verification were critical components of VERO's success.

Limitations and Future Work

The authors acknowledge a primary limitation: the underutilization of unlabeled data. VERO currently focuses on selecting a small, highly informative subset. Future work suggested includes:

  • Extending the approach to a semi-supervised framework to leverage more of the unlabeled data.
  • Exploring how VERO's principles can be applied to other multimodal tasks beyond MALSC.

Summary updated

BibTeX
@inproceedings{sun2025vero,
  author = {Kai Sun and Hao Wu and Bin Shi and Samuel Mensah and Peng Liu and Bo Dong},
  title = {{VERO: Verification and Zero-Shot Feedback Acquisition for Few-Shot Multimodal Aspect-Level Sentiment Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34707},
}
Figure