Whose Sarcasm? Separating Platform, Dialect, and Annotator Effects in Arabic Sarcasm Detection

Arabic sarcasm detectors are trained and scored almost entirely on tweets. Third-party annotators label those tweets as sarcastic or not, and the best systems plateau around 0.62 F1 on the sarcastic class. The usual reading of that ceiling is that sarcasm is hard and some dialects are harder. We think the number mixes three things that have never been pulled apart. The first is real model failure on a dialect. The second is label noise from annotators judging irony in a dialect or culture that is not their own. The third is the narrow slice of Arabic speakers who post on Twitter in the first place. This proposal builds a small, controlled evaluation set that varies all three factors on purpose. Texts come from two platforms with different user bases, four dialect groups, and two kinds of annotators: those from the writer's own dialect region and those from outside it. A subset also carries author-supplied "I meant this sarcastically" labels together with author demographics.

Two sentences for a colleague: we collect Arabic sarcasm data so that platform, dialect, and annotator origin can each be varied independently, then measure how much of today's detector error comes from each. We then test whether training on labels from same-dialect annotators improves detection of what authors actually intended on platforms the model never saw.

Core Claim

We claim that a substantial share of the per-dialect error measured on the standard Arabic sarcasm benchmark is caused by the labels, and that the remainder is dominated by platform shift rather than by dialect. Concretely, we predict three things. First, the per-dialect ranking of detector F1 will change depending on whether the gold labels come from same-region or out-of-region annotators. Second, the drop in F1 when a Twitter-trained detector is moved to a second platform will exceed the in-platform gap between dialects. Third, training on labels from same-dialect annotators will improve agreement with author-intended sarcasm across platforms more than the usual fixes do. Those usual fixes are rebalancing the under-represented dialects, adding a dialect-prediction head, or translating everything into Modern Standard Arabic first.

Every one of these predictions can fail. Each failure would still be informative, because each one tells the field which of the three factors to spend effort on.

What Exists and What It Leaves Open

The benchmark this work is built around is ArSarcasm-v2, introduced in the Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in Arabic. It contains 15,548 tweets labeled for sarcasm (19.2% positive), sentiment, and coarse dialect: Egyptian, Gulf, Levantine, Maghrebi, and Modern Standard Arabic. Fine-tuned AraBERT and MARBERT dominated the shared task, and the winning team reached 0.6225 sarcastic-class F1. The organizers themselves note that Maghrebi is under-represented. The set is Twitter-only and carries no author or annotator metadata.

A typical strong entry, ArSarcasm Shared Task: An Ensemble BERT Model for SarcasmDetection in Arabic Tweets, combines AraBERTv02 with Sentence-BERT embeddings and converts emoji to text. It reaches 59.89 F1, but it reports no per-dialect breakdown. Both papers motivate this proposal: they establish the ceiling without explaining it. We reuse their data and models. We do not reuse their assumption that the tweet labels are ground truth.

The idea that labels themselves are part of the problem comes from iSarcasm: A Dataset of Intended Sarcasm. That work had authors label their own posts and showed that models trained on third-party labels miss intended sarcasm. Exploring Author Context for Detecting Intended vs Perceived Sarcasm then showed that an author's posting history helps recover intent. Both are English-only and single-platform. They supply our method for collecting intended labels, and they motivate our question of whether the gap between intended and perceived sarcasm grows across dialect boundaries. Neither paper can answer that question.

The platform side is supported by several case studies, though none involves Arabic figurative language. Tweeting AI: Perceptions of Lay versus Expert Twitterati shows that Twitter over-samples younger, tech-literate users. The rJokes Dataset: a Large Scale Humor Collection acknowledges that its humor targets reflect Reddit's audience. OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs shows that models tuned on a single forum absorb that forum's subculture, and that demographic attributes leak into one another. That last finding constrains our design: we cannot treat "platform" as a clean stand-in for "demographics." This is why we add a self-reported author subset.

The closest methodological neighbor is Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairness. It adds a dialect head to a bias classifier so the model stops reading African-American English as offensive, and reports a 9.6-point F1 gain on detecting whether a post targets a group. Its dialect labels are silver, meaning a classifier predicted them, and it accepts its task labels as given. We use its multi-task design as a baseline. Our proposal differs from it on two points: it questions the task labels themselves, and it holds platform fixed or varies it on purpose. The concern that Arabic sarcasm often borrows offensive dialect vocabulary, raised by Leveraging Offensive Language for Sarcasm and Sentiment Detection in Arabic, is one specific way annotators from outside a region could misread irony.

Finally, the alternative of normalizing dialect into Modern Standard Arabic is represented by AraT5-MSAizer: Translating Dialectal Arabic to MSA. It reaches 21.79 BLEU on the OSACT 2024 test set and is trained largely on travel-phrasebook parallel data from MADAR, described in ST MADAR 2019 Shared Task: Arabic Fine-Grained Dialect Identification. No one has checked whether this kind of normalization preserves sarcasm, so we include it as a competitor.

Research Program

The evaluation set. We will build a factorial evaluation set. The first factor is platform: Twitter/X, plus a second public platform whose Arabic-speaking audience plausibly differs, namely comments on public Facebook pages and YouTube videos in the news and entertainment genres. The second factor is dialect group: Egyptian, Gulf, Levantine, and Maghrebi. The target is roughly 1,000 items per platform-by-dialect cell, about 8,000 items in total.

Sarcasm is rare, so pure random sampling would yield too few positives. Sampling only items that a Twitter-trained classifier flags would build Twitter's idea of sarcasm into the test set. We therefore draw each cell half at random and half from items a seed classifier flags. Every analysis is reported on the random half alone as well as on the full cell.

Annotation. Every item receives three labels from annotators born and raised in the item's dialect region and three from annotators raised in a different region. Annotators also give a confidence score and a short free-text cue explaining their judgment. This lets us estimate an agreement ceiling for each cell and inspect what out-of-region annotators miss.

The author-intended subset. Following iSarcasm, we recruit Arabic-speaking authors through a survey panel stratified by country, age band, and gender. Each author donates posts they wrote on either platform and labels their own intent. This yields roughly 1,500 posts with intended labels and self-reported demographics. It is the only part of the data where the demographic effect can be separated from the platform effect. Sample sizes per demographic group will be small, so we will report confidence intervals rather than point claims.

Analysis. We fit a simple decomposition of detector error into contributions from platform, dialect, and label source. In practice this is a mixed-effects logistic model of whether a detector's prediction is correct. Platform, dialect, and annotator origin enter as fixed effects, and item enters as a random effect. Fitting it on the random half of each cell gives an estimate that does not depend on how candidates were flagged.

The training intervention. We test one training change. Fine-tune MARBERT on ArSarcasm-v2 and continue training on a disjoint pool of Twitter items labeled by same-dialect annotators, using soft labels. Compare this against the same continued training on out-of-region labels. Both training pools have the same size and come from the same tweets, so annotator origin is the only thing that differs.

Baselines

The naive reference is a majority-class predictor plus a TF-IDF logistic regression over word and character n-grams. The second is the surface-feature family that won MADAR dialect identification, where Multinomial Naive Bayes reached 62.12% macro F1 on the test set according to ST MADAR 2019 Shared Task. Including it tells us whether any cell is solved by dialect-specific surface markers alone.

The strongest existing method is MARBERT or AraBERT fine-tuned on ArSarcasm-v2 in the configuration of the top shared-task systems. The reported benchmark number is 0.6225 sarcastic-class F1 by BhamNLP, from the WANLP 2021 overview. We also run the AraBERTv02 plus Sentence-BERT ensemble from Bashmal and AlZeer's shared-task paper, which reports 59.89. We will first reproduce a number within about two points of 0.6225 on the official ArSarcasm-v2 test split before trusting any comparison. All models get the same training data, the same search over learning rate and epochs (twenty trials each), and five random seeds.

The dialect-aware multi-task baseline adapts the dialect-disentangling design. It adds a dialect head trained on ArSarcasm-v2's gold dialect labels, together with loss reweighting that up-weights Maghrebi. This is the most direct competitor to our training intervention, because it targets the same per-dialect gap.

The normalization pipeline passes every input through AraT5-MSAizer (21.79 BLEU on OSACT 2024 Task 2, per Fares) and then classifies with the same MARBERT detector fine-tuned on normalized training data.

The prompted large language model is a strong instruction-tuned multilingual model. We give it dialect-labeled few-shot examples drawn from ArSarcasm-v2 and tune the prompt on the development split only.

The key ablation isolates annotator origin. It is the continued-training comparison above: same-dialect versus out-of-region labels, with identical items and identical compute.

What counts as beating the baselines. The intervention wins if it beats both the strongest fine-tuned baseline and the multi-task baseline on author-intended F1 on the non-Twitter platform. The margin must exceed the seed-to-seed spread and hold up under a paired bootstrap test at p < 0.05.

Why not just…?

Why not just fine-tune MARBERT and report per-dialect scores? We do exactly that as the main baseline. The problem is what a per-dialect score on ArSarcasm-v2 means. It is computed against labels whose annotators' regions are not reported, on a single platform, so a low Maghrebi score could reflect the model, the labels, or the sample. The comparison of in-region and out-of-region labels on identical items is the experiment that tells these apart. If the per-dialect ranking does not move, that is evidence the plain approach is measuring what it claims, and we will say so.

Why not just translate dialect into Modern Standard Arabic and use one standard-register detector? The translator was trained mostly on travel phrases and scores about 22 BLEU. That makes it plausible that sarcastic cues such as tone particles, deliberate misspellings, and ironic honorifics get flattened. Still, nobody has measured this, and normalization could help by reducing vocabulary sparsity. We measure it directly in two ways. We run the pipeline as a baseline. We also have same-dialect annotators relabel 500 translated items, which tells us whether sarcasm survives translation regardless of how any classifier performs.

Why not just rebalance the dialects or add a dialect head, as in the English dialect-fairness work? This is the most serious competitor for closing the in-platform gap, and it may well close it. What it cannot do is repair labels or recover from platform shift. If the multi-task baseline matches our intervention on the second platform and on author-intended labels, then annotator origin does not matter, and our training claim fails. The head-to-head comparison under matched budgets settles this.

Why not just prompt a large language model? Theory-of-mind evaluations such as CoMMET report real limits in tracking non-literal intent. Prompted models may also carry their own platform priors, as OpinionGPT suggests. We include one as a baseline. A prompted model that generalizes across platforms better than fine-tuned encoders would be an important finding in its own right, because it would point to fine-tuning on tweets as the source of platform overfitting.

Proposed Experiments and Decisive Checks

Check 1: label effect. For each detector and dialect, compute sarcastic-class F1 against majority-vote labels from same-region annotators and from out-of-region annotators, on the Twitter cells. Report Krippendorff's alpha within each annotator group, and the agreement between the two groups. The label-effect claim holds if out-of-region agreement with authors' intended labels is lower than same-region agreement, and if detector rankings across dialects change between the two label sets. If agreement between the groups is as high as agreement within them, the claim is refuted.

Check 2: platform shift versus dialect gap. For Twitter-trained detectors, compare two quantities against same-region labels. One is the F1 drop from Twitter to the second platform, averaged over dialects. The other is the largest gap between dialects within Twitter. The mixed-effects model gives these with intervals. We report results on the random half to rule out the seed classifier as an explanation. The author subset then tells us how much of any platform effect survives after controlling for authors' self-reported age band and country.

Check 3: normalization. Compare F1 of the normalization pipeline against the raw-text detector in every cell. Add the human relabeling of translated items: the fraction of items that are sarcastic in the original but read as sincere after translation measures pragmatic loss directly.

Check 4: the intervention. Compare MARBERT continued on same-region labels against continued on out-of-region labels, and against the multi-task baseline, the ensemble, and the prompted model. The primary metric is sarcastic-class F1 against author-intended labels, pooled over platforms and reported per cell, with 95% bootstrap intervals. A secondary check is that ArSarcasm-v2 test F1 should stay within about two points of the reproduced 0.6225-level baseline, so that any gain is not bought by forgetting the original task.

Taken together, these checks turn the 0.62 ceiling into a breakdown of where Arabic sarcasm detectors actually fail, and they produce the first Arabic sarcasm set where platform, dialect, and annotator origin can each be varied independently.