A common way people push back against hate online is to repost it. A user screenshots an abusive post, embeds it in their own message, and writes "this is disgusting" above it. To a content filter, the slur is right there in the pixels or the text it reads off the image, so the person condemning the abuse gets flagged as its author. Philosophers call the underlying distinction use versus mention. Using a word deploys its meaning, as when someone hurls a slur. Mentioning a word treats it as an object of discussion, as when someone quotes the slur to condemn it. In screenshot-style posts, the evidence for which one is happening is visual: a card border, a different avatar, a nested layout. Text extraction that reads characters off the image erases exactly this evidence before the classifier ever sees it.
We propose to recast use-versus-mention in visually framed posts as a speaker attribution problem: for each harmful phrase, decide who is saying it. We then teach a small, deployable model to solve it. A large teacher model, prompted with use-mention demonstrations, writes a short structured record for each post. The record lists each harmful span, which visual frame it sits in, whose voice that frame represents, and whether the poster endorses or rejects it. A compact student is trained to produce this record before its label. The bet is that this short attribution record, not free-form chain-of-thought and not the label alone, is the piece of reasoning that removes counter-speech false positives at small model sizes.
Core Claim
Gligoric et al. (NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps) showed that prompting GPT-4 to reason explicitly about use versus mention before classifying cuts its false-positive rate on hate-speech counter-speech by 82.6%, at a 2–3% cost in true positives. This work motivates ours and also sets the bar. It is text-only, it depends on a very large closed model, and it found that quotation marks, the obvious textual cue for mention, make filters worse. We read that last result as evidence that the mention signal must be represented structurally, as "this span belongs to someone else," rather than as a surface token.
Our claim is that, for posts where harmful text appears inside an embedded visual frame, a student under roughly 1B parameters that first emits a frame-grounded attribution record can come close to the counter-speech false-positive rate of the prompted teacher. It would do so while keeping the teacher's true-positive rate. It would also clearly beat two alternatives at equal training data: a student distilled on labels alone, and a student distilled on unconstrained rationales.
The closest distillation work is HARE (HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning). HARE distills teacher rationales for implicit hate into Flan-T5 and reports up to 7.7 F1 of cross-dataset gain. It supplies our training recipe. It never evaluates counter-speech, uses only text, and its rationales are free prose. We differ on all three counts. The structured, short target also addresses the verbosity and rationale–label mismatch that the reasoning-distillation literature flags.
Research Program
The first component is a controlled benchmark of framed counter-speech. We take the 1,826 harmful-statement and counter-speech pairs from Gligoric et al. and render each as image posts in several layouts: a quote-post card, a reply screenshot, a screenshot of a chat, and a caption over an image macro. In each layout the harmful statement sits in an embedded frame and the counter-speech is the poster's own text.
For every rendered item we also build minimal "frame-swap" twins in the spirit of the benign confounders in The Hateful Memes Challenge. A frame-swap twin keeps identical characters but moves the harmful statement out of the embedded frame into the poster's own caption. A stance-flip twin keeps the frame but makes the caption endorse the statement ("finally someone says it"). A model that truly tracks attribution must change its label across these twins. A model relying on word shortcuts will not.
The second component is the teacher pipeline, which runs offline. A layout-aware extractor recovers text regions and their nesting. OmniParser (OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition) supplies the method here: it jointly predicts text locations and structural tokens, which gives us a region hierarchy rather than a flat string. Its heavy compute (it runs at 1.3 frames per second) is acceptable here because it runs only when building training data.
A large teacher then writes the attribution record. It is prompted with the extracted regions plus a handful of worked use-mention demonstrations, in the in-context style that Gligoric et al. found effective. Following the "solution-in-prompt" trick of Teaching Small Language Models to Reason, where the gold answer is put in the teacher's prompt so its rationales stay consistent, we condition the teacher on the gold label. We discard records whose attribution contradicts the known rendering, since for synthetic items we know which frame each span is in.
The third component is the student. We train a compact vision-language student, and a text-only student, to output the attribution record followed by the label. The text-only student reads extracted text with explicit frame tags. Real data come from two sources. The first is the counter-speech class in Botelho et al.'s re-annotation of MMHS150K (Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal Hate). The second is CAD (Introducing CAD: the Contextual Abuse Dataset), with its collected but never-used post images reattached. Both are used for training and for held-out testing on real posts.
In-context learning plays two roles. It is how the teacher is elicited. It is also a cheap competitor, examined below.
Baselines
The naive reference point is a fine-tuned discriminative classifier on the same real data. On CAD this means BERT, for which the original paper reports macro F1 of 0.455 and counter-speech F1 below 0.1. On MMHS150K it means Botelho et al.'s late-fusion RoBERTa+Xception model, which reports overall F1 of 0.771. These numbers are reported overall or per-class, not as counter-speech false-positive rates, so we rerun both models on our splits rather than borrow their figures.
The strongest existing method is the prompted teacher itself: GPT-4 with use-mention mitigation and chain-of-thought, following Gligoric et al. They report a baseline false-positive rate of about 9% on hate-speech counter-speech and an 82.6% relative reduction with the intervention. We run that exact prompt on the text of our rendered items. We also run a version that receives the extracted region hierarchy, so the teacher is not handicapped by missing layout.
Beating the naive baselines means a large cut in counter-speech false positives at matched true-positive rate. Against the teacher, success is not beating it. Success is retaining most of its false-positive reduction, which we set as at least two-thirds, at a small fraction of its inference cost and with true-positive loss no larger than its 2–3%.
The ablations that isolate our key ingredient all use identical student architectures, identical teacher-labeled items, and the same tuning budget. The first is a student trained on teacher labels only. The second is a HARE-style student trained on free-form teacher rationales. The third is our student trained on the attribution record. A fourth arm removes frame information from the record and keeps only span and stance. This tests whether attribution to a frame, specifically, is what matters.
For general robustness we also report the HateCheck counter-speech functional tests (HateCheck: Functional Tests for Hate Speech Detection Models), where Perspective API's aggregate accuracy is reported at 76.6%. No reported counter-speech-specific number was found for it on Lacuna.
Why not just…?
Why not just distill the teacher's labels into a small classifier? This is the most serious threat. Teaching Small Language Models to Reason found rationale distillation helps most on multi-step tasks and least on knowledge-heavy ones. Use-versus-mention might be shallow enough that labels suffice. Our evidence for rationales is indirect: HARE's out-of-domain gains and CAD's collapse of supervised counter-speech detection. The label-only arm settles the question directly. If it matches our student on the frame-swap and real-data tests, we will report that the structured record is unnecessary.
Why not just read the text off the image, wrap the embedded frame in explicit tags, and use a text model? This is also a real threat, and we keep it as a full arm rather than a straw man. Gligoric et al.'s quotation-mark result suggests that surface markers can backfire. But speaker tags such as "[embedded post by another account]" are not quotation marks and may work. If the tagged-text student matches the vision-language student, the contribution becomes the attribution target and a cheap tagging front-end, and the vision-language student is dropped. The vision-language student earns its place only if it wins on layouts where frame detection by rules fails, such as chat screenshots and image macros. We report results per layout to see this.
Why not just steer a mid-size model with a use-mention task vector? In-context learning compresses demonstrations into a single internal vector that can be patched into a query-only pass; In-Context Learning Creates Task Vectors shows this recovers 80–90% of few-shot accuracy on single-token tasks. Our final label is a single token, so this is cheap and plausible. We extract a task vector from use-mention demonstrations on a 7B model and evaluate it on the same splits. We expect it to fail on frame-swap twins, because the original vector work covered simple rules. Here the label depends on structure in the input rather than a fixed mapping. That expectation is a hypothesis, not a finding.
Why not just run the large model at moderation time? The cost of multi-step decoding at platform volume is the gap Gligoric et al. leave open, and real-world performance is already four times worse than benchmarks suggest (HateDay). We report latency and cost per thousand posts for every arm so the trade-off is explicit.
Proposed Experiments
The decisive check is the frame-swap test. On twins with identical characters, we measure how often each model's label flips when the harmful statement moves between the embedded frame and the poster's own text. We report counter-speech false-positive rate and harmful-content true-positive rate separately, broken down by targeted identity group, since Gligoric et al. found Jewish people and people of color are most affected.
The second check is transfer to real posts. We train on rendered data plus part of the real data, then test on held-out MMHS150K counter-speech and CAD counter-speech items with images. We report counter-speech F1 against the rerun baselines. This answers whether synthetic rendering teaches attribution or teaches templates. A student that aces rendered twins but does not improve on real counter-speech has learned the templates.
The third check is a data-efficiency curve at 10%, 25%, 50%, and 100% of teacher-labeled items. Each training method (labels only, free rationales, attribution record, record without frames) gets one curve. We test whether structured supervision shows the kind of efficiency gap Magister et al. found for arithmetic.
Finally, we audit inherited teacher error. Two annotators label attribution on 300 real posts, and we measure how often the student copies teacher mistakes. This addresses the bias-inheritance risk the HARE line of work raises.
References
- NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps
- HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
- OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition
- Teaching Small Language Models to Reason
- Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal Hate
- Introducing CAD: the Contextual Abuse Dataset
- HateCheck: Functional Tests for Hate Speech Detection Models
- In-Context Learning Creates Task Vectors
- HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter