When a social-media post says "the new phone looks great but the battery is a joke" and comes with a photo, a model doing aspect-based sentiment analysis must do three things. It must find each target being judged ("phone," "battery"). It must link each target to the words and image regions that judge it. It must then decide whether each judgment is positive, negative, or neutral. Specialized models trained on Twitter benchmarks still beat general multimodal assistants at this. CORSA reaches 81.1% accuracy on Twitter-2015 sentiment classification, while zero-shot GPT-4V reaches 75.3%, both reported in Multimodal Aspect-Based Sentiment Analysis under Conditional Relation. Nobody knows which of the three steps that gap lives in. Nobody knows what the specialist gives up in general ability to get there either.
This note proposes to answer both questions with one tool: an evaluation that scores each step separately, built mostly from labels that already exist. The diagnosis then drives an intervention. If the generalist's deficit sits in one step, typically linking an opinion to the right target, we fine-tune on supervision for that step alone rather than on final sentiment labels. The hypothesis is that this recovers most of the specialist's accuracy while paying a much smaller "specialization tax," the loss of general skills that follows narrow fine-tuning. The reason to expect this is that end-label training also teaches the model dataset shortcuts, such as entity names that usually appear with positive sentiment, and those shortcuts are exactly what disturbs general behavior.
Core Claim
Our claim has two parts, and both can be falsified. First, a tiered diagnosis of multimodal aspect sentiment will show that the generalist-versus-specialist gap is concentrated in one tier rather than spread evenly. We expect that tier to be binding, meaning the correct pairing of each aspect with its own evidence when a post carries several aspects with conflicting sentiment. The tiers are: Tier 1, finding aspects and judging whether the image is relevant; Tier 2, binding; and Tier 3, polarity given correct inputs. Second, fine-tuning an open multimodal model only on supervision for the failing tier will match end-label fine-tuning on the benchmark within a small margin, and will lose clearly less on general vision-language benchmarks than either full fine-tuning or importance-masked fine-tuning on end labels.
If the first part fails because errors are spread evenly, the second part loses its rationale, and we report a negative result on tier-targeted specialization. If the gap instead turns out to sit in Tier 1 image relevance, the conclusion is that a CORSA-style relevance gate is the right fix, and we say so.
How This Builds on Prior Work
Three groups of papers motivate the question.
The multimodal sentiment systems AoM, CORSA, PTA and AETS all report only end-to-end scores. AoM (AoM) uses aspect-aware attention and a syntax graph. CORSA adds a gate that detects irrelevant images and a module that locates the visual object. PTA (PTA) is an extract-then-classify pipeline. AETS (AETS) masks aspects and prunes text using dependency parses. None of them asks where its improvement comes from, and none measures what fine-tuning costs elsewhere. Dual-path counterfactual work (DPCI) shows that aspect-to-label shortcuts exist in these datasets. That finding is evidence for our hypothesis about why end-label training damages general behavior, but DPCI does not separate the tiers.
Tiered evaluation elsewhere supplies the method and the expectation of a sharp drop at the linking stage. PhD found sentiment to be the hardest of its hallucination probes, and found that misleading text makes open models override what they see (LLaVA-1.5-L falls from 0.443 to 0.011). Plot Twist found models answering from chart titles rather than pixels. GEOTEMP found near-perfect single-step accuracy collapsing to 25.4% on joint multi-hop questions.
Two structured-prompting papers already decompose sentiment into steps. Syn-Chain prompts through syntax, then opinion, then polarity, and is text only. PanoSent uses a chain-of-sentiment over multimodal dialogue but never checks whether the visual target was located. We reuse their decompositions as probes and as baselines. We differ in scoring each step against labels and in using the result to decide what to train.
On specialization, SPIDER supplies both the evaluation protocol and the strongest anti-forgetting baseline. SPIDER updates only the parameters whose downstream gradient importance exceeds their pre-trained importance, and it scores retention with a harmonic mean of downstream and upstream accuracy. Its experiments cover captioning and VQA, not tasks that require several reasoning steps. Specializing Smaller Language Models towards Multi-Step Reasoning shows the general cost of narrow specialization at the 11B scale, which constrains our backbone choice to models where that cost is visible.
Research Program
The first component is the diagnostic, assembled on Twitter-2015 and Twitter-2017 together with CORSA's C-MABSA annotations of image relevance and bounding boxes.
Tier 1 is scored two ways. Aspect extraction uses F1 against the existing extraction labels. Visual relevance uses accuracy at deciding whether the image depicts the aspect, plus box overlap where boxes exist. The C-MABSA labels are silver, produced by YOLOv8 and UNINEXT, so we hand-verify a stratified sample of about 500 items and report Tier 1 against both the silver and the verified labels.
Tier 2, binding, needs no new annotation. We take posts with two or more gold aspects of differing polarity and ask the model about each aspect in turn. A post counts as bound correctly only if every aspect receives its own polarity. A model that copies one sentiment across the whole post fails this test even if its per-aspect accuracy looks reasonable. We add an image-swap control: we replace a relevant image with one from another post, and measure how often predictions change for aspects whose relevance label says the image matters, versus aspects whose label says it does not.
Tier 3 is polarity accuracy conditioned on Tier 1 and Tier 2 being given or correct. This isolates the synthesis step.
From these scores we compute conditional accuracies, for example Tier 3 accuracy given correct binding. These tell us what fraction of end-task error each tier accounts for.
The second component uses the diagnosis to answer the question from the research file about textual scaffolds. Does prompting a multimodal model to write a syntax-then-opinion chain, as Syn-Chain does, improve binding? Or does it increase the image-override effect that PhD found? We measure this directly as the change in image-swap sensitivity with and without the scaffold.
The third component is the intervention. We use one open backbone that SPIDER also used, VILA or LLaVA-1.5, so that SPIDER's protocol and code transfer directly. Every training run gets the same number of training examples and steps. The arms differ only in supervision:
- End-label arm: final aspect-polarity tuples.
- Tier-targeted arm: targets for the failing tier only. For binding, these are per-aspect contrastive pairs from the same post. For relevance, they are the C-MABSA relevance labels.
- Combined arm: tier-targeted supervision plus SPIDER masking.
General retention is measured on SPIDER's upstream suite (OKVQA, TextVQA, GQA, OCRVQA) and on the PhD benchmark. Within PhD we report its sentiment subset separately, because a sentiment specialist that gets worse at general visual sentiment would be an especially telling form of tax.
Baselines
The naive reference point is the untouched open backbone prompted zero-shot for aspect-sentiment tuples. We found no reported result on Lacuna for VILA or LLaVA-1.5 zero-shot on Twitter-2015 or Twitter-2017, so we measure it. This run also anchors the upstream scores from which all tax is computed.
The strongest general-model reference is GPT-4V or GPT-4o zero-shot. CORSA reports 75.3% classification accuracy for GPT-4V on Twitter-2015. We rerun it through the tiered diagnostic so that its deficit is located, not just sized.
The strongest specialized systems are CORSA and AETS, with numbers taken from their papers. CORSA reports 81.1% classification accuracy and 69.9% joint F1 on Twitter-2015. AETS reports joint F1 of 72.1% on Twitter-2015 and 73.1% on Twitter-2017. Where released code permits, we also run them through Tiers 1 to 3, which tells us whether their advantage is concentrated where we expect. Their accuracy figures define what "recovering most of the specialist's gain" means: the tier-targeted arm counts as competitive if it closes at least 75% of the gap between the zero-shot backbone and CORSA's 81.1% on Twitter-2015 classification. The 75% threshold is our choice.
The strongest competitor on the actual question, accuracy per unit of general loss, is SPIDER applied to end labels. On VILA with Flickr30k, SPIDER reports a harmonic mean of 66.61 against 55.16 for full fine-tuning, with OKVQA retention of about 47 against about 37. We run SPIDER, LoRA, and full fine-tuning on identical sentiment data with identical step counts. Each method gets the same hyperparameter search budget of eight configurations, selected on a held-out Twitter development split. The tier-targeted arm beats SPIDER if its harmonic mean of Twitter accuracy and upstream accuracy is higher across three seeds. An upstream drop no larger than SPIDER's, at comparable task accuracy, counts as a tie.
The ablation that isolates our key ingredient holds data and parameter budget fixed and changes only the labels. Binding-pair supervision is compared against end-label supervision on the same posts, and against binding pairs whose aspect-polarity assignment is shuffled. The shuffled control tests whether any gain comes from the contrastive pairing itself rather than from the extra exposure to multi-aspect posts.
Why not just…?
Why not just prompt GPT-4o zero-shot and accept the gap? The gap is real: 75.3 versus 81.1 on Twitter-2015 in CORSA, and more than 15 F1 points between ChatGPT or LLaMA-2 and PTA's pipeline. Prompting a frontier model also says nothing about where it fails. We include it as a baseline, and if the tiered diagnosis shows its deficit is mostly in image relevance, the cheap fix is a relevance gate in front of it, a result we would report as such.
Why not just use a structured prompt like Syn-Chain or Chain-of-Sentiment? Both helped over unstructured reasoning in text or dialogue. However, PhD and Plot Twist show multimodal models letting text override images, and a written syntactic scaffold may make this worse. This alternative is a genuine threat on accuracy, so we run it as a full baseline and measure its image-swap sensitivity. If it closes the gap without fine-tuning and without increasing image override, it wins outright, because it pays no specialization tax at all.
Why not just apply SPIDER to end labels? This is the most serious alternative. SPIDER already improves retention substantially on VILA, and it may be enough. Our argument is that masking decides which parameters move, while tier-targeted supervision changes what the model is taught. If end labels teach entity-polarity shortcuts, masking cannot filter them out. The combined arm and the head-to-head comparison settle this. If SPIDER on end labels matches the tier-targeted arm on both axes, we drop the claim that supervision choice matters.
Why not just caption the image and run text-only sentiment analysis? GeoSQA found that even human-written transcriptions of visual content do not fix reasoning failures. CORSA reports that many Twitter images are irrelevant to the aspect, so captions inject false cues such as a smiling face beside neutral text. We run caption-then-Syn-Chain as a cheap baseline. Its Tier 2 binding score shows directly whether converting the image to text helps or hurts linking.
Decisive Checks
Three results decide the proposal.
The first is the error decomposition. If no tier accounts for more than half of the gap between the open backbone and CORSA, the targeted-specialization premise fails, and the paper becomes a diagnostic-only contribution.
The second is the scaffold check. If Syn-Chain-style prompting raises binding accuracy and does not raise image-swap sensitivity on relevance-negative items, structured prompting should be preferred over fine-tuning.
The third is the matched-budget comparison across three seeds: tier-targeted training versus SPIDER on end labels, scored by task accuracy, the upstream harmonic mean, and the PhD sentiment subset. The proposal succeeds only if tier-targeted training improves retention at comparable accuracy, and only if the shuffled-pair ablation removes that improvement.
References
- Multimodal Aspect-Based Sentiment Analysis under Conditional Relation
- AoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis
- PTA: Enhancing Multimodal Sentiment Analysis through Pipelined Prediction and Translation-based Alignment
- Aspect Enhancement and Text Simplification in Multimodal Aspect-Based Sentiment Analysis for Multi-Aspect and Multi-Sentiment Scenarios
- Dual-Path Counterfactual Integration for Multimodal Aspect-Based Sentiment Classification
- PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Around the World in 24 Hours: Probing LLM Knowledge of Time and Place
- Aspect-Based Sentiment Analysis with Syntax-Opinion-Sentiment Reasoning Chain
- PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis
- Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning
- Specializing Smaller Language Models towards Multi-Step Reasoning
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level