Dialect-Written Label Descriptions for Rare Classes

Classifiers that detect rare language phenomena such as sarcasm or irony usually learn the rare class from a few hundred examples. Those examples almost always come from the standard variety of the language. When the classifier is deployed on a regional variety, for instance Indian English instead of British English, it fails in two ways at once. It has seen too few examples of the rare class, and the examples it has seen signal that class with the wrong cues. We propose replacing the rare class's learned weight vector with a short natural-language description of the phenomenon, written by a speaker of the target variety, that says how sarcasm sounds in that variety. The model then decides whether a text matches that description. The bet is that a paragraph of expert description can stand in for many labeled examples of the rare class, and that it can be swapped per dialect without retraining.

The claim is narrower than "label descriptions help," and it can be falsified. A description gives the rare class a representation whose quality does not depend on how many training examples that class has. Its advantage over an ordinary classification head should therefore grow as the class gets rarer and as the test dialect moves further from the training dialect. If the gain is flat across rarity and dialect, descriptions are just a generic regularizer and the proposal fails. We also measure an exchange rate. At equal annotator time, is a dialect-specific description worth more than labeled target-dialect examples?

What exists and what is missing

Label-description methods recast classification as matching a text against natural-language category descriptions. Description Based Text Classification with Reinforcement Learning concatenates a description with the input in a reading-comprehension format and learns descriptions with reinforcement learning. It cut BeerAdvocate error from 27.8% to 15.6%, but its training was high-variance. Description-Enhanced Label Embedding Contrastive Learning for Text Classification (DELE) attaches WordNet glosses to labels and trains with attention in both directions between sentence and label, plus a contrastive loss. The Benefits of Label-Description Training for Zero-Shot Text Classification (LDT) fine-tunes on dictionary and encyclopedia descriptions alone and reports gains of 17.5 to 18.9 points in zero-shot accuracy. We reuse the reading-comprehension input format from these papers. All three, however, draw descriptions from standard-English dictionaries, and all test on roughly balanced topic and sentiment data. They say nothing about rare classes or regional varieties.

A second line of work motivates the gap. BESSTIE benchmarks sentiment and sarcasm across Australian, British and Indian English. Its companion report, BESSTIE, finds three things:

  • sarcasm F1 "typically ranging between 0.50 and 0.60";
  • Indian English averaging 0.63 F1 on Google reviews;
  • sharp drops when models are fine-tuned on one variety and tested on another.

Every BESSTIE model uses a plain classification head. UO-CERPAMID at IroSvA detects irony in Spanish from Spain, Mexico and Cuba using bag-of-words features, and reports 0.5930 macro-F1 on Cuban Spanish. It found that stylistic features such as repeated punctuation behaved as noise. Variation and Instability in Dialect-Based Embedding Spaces explains why standard-English descriptions may transfer poorly. Nearest-neighbor overlap between European and North American English word spaces is only 26.8%, against about 74% within one variety. Drift is worst for abstract, socially conditioned words, which is exactly the vocabulary a description of sarcasm uses.

Two imbalance papers constrain the design:

  • PINGAN Omini-Sinitic raised multi-label F1 for patronizing-language detection from 36.95 to 47.29 using category-name prompts, label attention and oversampling, where only 9.5% of samples are positive. This is the closest prior method, but its labels are bare names like "Compassion" and its English is treated as one variety.
  • PALI-NLP found that a supervised contrastive loss slightly hurt rare sarcasm subtypes relative to adversarial training alone. For that reason we do not pull minority examples toward each other. Instead we anchor them to a fixed description.

Core Claim

The proposed model scores a text $x$ against a class description $d_c$ with a cross-encoder, $s(x, d_c)$. A cross-encoder here is a single transformer that reads the description and the text together. It is trained with class-weighted cross-entropy, and the rare class's description is conditioned on dialect ($d_{c,v}$ for variety $v$). The majority class keeps a generic description. Changing the target dialect means changing a paragraph of text, not retraining a head.

We predict that the F1 gain over an index-based head, $\Delta(\pi, v)$, rises as $\pi$ falls and rises with the distance of $v$ from the training varieties. Here $\pi$ is the rare-class rate in the training data and $v$ is the test variety. We also predict that dialect-written descriptions beat generic ones only on shifted varieties.

Research Program

We build three description sets for "sarcastic" or "ironic."

The first is a generic set in the style of LDT, made from dictionary definitions and synonyms.

The second is dialect-written. Two native-speaker annotators per variety each write a description of about 150 words of how sarcasm or irony is typically signaled in their variety, with a few illustrative phrases. They get no access to test data, and their time is logged in minutes. Examples of what they might describe are code-mixed exaggeration and regional idioms of the kind BESSTIE notes for Indian English.

The third is written by a large language model when prompted with the variety name. This tests whether expert writing is needed at all or can be automated. Hallucinated dialect features are a known risk here.

For the backbone we use a RoBERTa-large encoder and a multilingual encoder for Spanish, so the comparison with BESSTIE's encoders is direct. To control rarity, we subsample the rare class in the training split to its natural rate and then to one half, one quarter and one eighth of it. Test sets keep their natural distribution.

Dialect shift has two settings:

  • English: train on British and Australian English, test on Indian English, using BESSTIE's Reddit and Google-review sarcasm splits, roughly 3,000 Reddit comments per variety.
  • Spanish: train on Spain Spanish, test on Mexican and Cuban Spanish, using IroSvA 2019.

In-variety test sets serve as the no-shift control. Every configuration runs with five seeds, because the RL description work and PALI-NLP both report sensitivity to seeds and hyperparameters.

Baselines

The naive reference point is the setup BESSTIE reports. We fine-tune the same encoder with a standard classification head and unweighted cross-entropy on the source varieties, then apply it directly to the target variety. BESSTIE's reported sarcasm F1 of 0.50 to 0.60 frames what "no intervention" looks like. We re-run it on our exact splits.

The imbalance-only competitor is the same head with class-weighted cross-entropy and minority oversampling. PALI-NLP's re-weighted BERTweet reached 0.1564 macro-F1 on fine-grained sarcasm, which shows this baseline is weak on very rare classes but still standard.

The strongest existing method is a reimplementation of PINGAN Omini-Sinitic's approach on our data: category-name prompts, label attention, and oversampling. It reported 47.29 average F1 on the patronizing-language data. There is no reported result on Lacuna for it on BESSTIE or IroSvA, so we run it ourselves. It gets the same tuning budget as our model: an equal number of hyperparameter trials and the same training steps.

Beyond these, we include a generic-description cross-encoder in the style of LDT and DELE. It is identical to our model except for the description text, so it isolates the dialect ingredient.

We also run a placebo ablation. The description is replaced by a length-matched paragraph describing an unrelated register. This checks that gains come from content and not from extra input tokens.

The most important baseline is matched-cost target-dialect fine-tuning. We give the index-head model as many labeled target-variety examples as the description writers' logged minutes would buy at a separately measured labeling rate.

We count the proposal as winning if it meets two conditions over five seeds: it beats the PINGAN-style model and the matched-cost fine-tuning on rare-class F1 for the shifted variety, and the interaction predicted in the Core Claim appears.

Why not just…?

Why not just prompt a large language model with a definition of sarcasm? BESSTIE reports that older encoders often beat newer, larger decoders on these tasks. LDT reports that RoBERTa-large trained on descriptions beat GPT-3.5 on AGNews and DBPedia. Neither result concerns rare classes under dialect shift, so this remains a real threat. We will run instruction-tuned decoders zero-shot with both the generic and the dialect-written descriptions. If the decoder with dialect descriptions matches our encoder, the contribution collapses to the descriptions themselves. We would report that as the finding, and it would still be useful because the artifact transfers.

Why not just reweight or oversample? Reweighting changes how much each rare example counts but not what the model learns from so few of them. It also cannot supply dialect cues that are absent from source-dialect data. PINGAN's own gains came from adding label semantics on top of oversampling. The imbalance-only baseline, run at each subsampled rarity level, will show directly whether reweighting closes the gap as $\pi$ shrinks.

Why not just label a few target-dialect examples? This is the most serious alternative, and the exchange-rate experiment exists to settle it. If a few dozen labeled Indian English sarcasm examples, at matched annotator time, outperform the description, we will say so. The proposal would then reduce to a cheaper method when no target labels exist at all.

Why not use contrastive alignment as DELE does? PALI-NLP found that supervised contrastive loss hurt rare sarcasm subtypes. We therefore run DELE's contrastive term as an optional add-on. We report whether it helps once minority examples are anchored to a fixed description rather than to each other. This speaks directly to the contradiction between the two papers.

Decisive Checks

We report rare-class F1 and macro-F1 on each test variety, recall at fixed precision, and the gap between predicted and true positive rate as a measure of drift toward the majority class. Four outcomes decide the proposal:

  1. The description gain grows as the training positive rate is cut from natural to one eighth. We test this with a seed-level regression of $\Delta$ on $\log \pi$ and on shifted versus unshifted test variety.
  2. Dialect-written descriptions beat generic ones on Indian English and on Cuban and Mexican Spanish, but not on the source varieties.
  3. The placebo description gives no gain.
  4. The exchange rate at matched annotator time is above one.

If outcome 2 holds only for human-written and not for LLM-written descriptions, then the expert paragraph is the key ingredient. We would quantify its cost.