Online moderation data is mostly benign. The harmful cases are rare, and the finest-grained harmful categories are rarer still: insults aimed at a group rather than an individual, or abuse aimed at an organization. A large language model (LLM) used as an automated grader, or "LLM judge", looks like a natural fit for these hard cases because it can read context and follow a written policy. The evidence says otherwise when the LLM is used alone. Zero-shot LLMs trail small fine-tuned classifiers on the standard offensive-language benchmarks, and the gap is widest in the sparse sub-categories. The reliability machinery that exists for LLM judges also controls the wrong quantity. It bounds the share of wrong verdicts among the verdicts the system accepts. When 95% of items are benign, a system that calls everything benign satisfies that bound and misses nearly every attack.
This note proposes to recast the problem as a budgeted guarantee. The system must provably catch at least a fraction $1-\beta$ of each rare offense class, for example group-targeted insults. It may send items it is unsure about to human reviewers. The goal is to use as few human reviews as possible. We claim that this guarantee can be enforced with class-conditional calibration: thresholds set using only examples of the rare class itself. We further claim that the guarantee survives changes in how common offensive content is between calibration and deployment, which a false-discovery guarantee does not. The empirical bet is narrower and can be falsified. A cascade, in which a fine-tuned encoder handles the confident bulk and an LLM judge re-scores only the uncertain slice, should need fewer human reviews than either model alone at the same guaranteed recall. If the encoder alone does as well, the LLM judge has no place in imbalanced moderation, and we will report that result.
Core Claim
Two statements need to be kept apart.
The first is statistical, and we expect it to hold by construction. Let $s(x)$ be any score for "item $x$ belongs to rare class $k$." We pick the threshold $\tau_k$ using only calibration items whose true label is $k$, and we set it so that the fraction of such items scoring below $\tau_k$ is at most $\beta$ with confidence $1-\delta$ (via a Clopper–Pearson bound). Then the miss rate on class $k$, conditional on the item truly being class $k$, is bounded. This conditioning on the true class is the key property. The bound depends only on the score distribution within class $k$, so it is unaffected when the base rate of class $k$ changes. A false discovery rate, the error rate among accepted verdicts, mixes in the base rate and does not share this property.
The second statement is empirical and uncertain: whether an LLM judge is a better scorer than a fine-tuned encoder on exactly the ambiguous region. "Better" here has a specific meaning. At a fixed guaranteed per-class recall, the better scorer sends fewer items to humans. We measure human-review fraction at guaranteed recall instead of macro-F1, because macro-F1 is where LLMs are already known to lose and it does not reflect how moderation systems are actually run.
Where This Sits
Judge, Retrieve, or Abstain supplies the method template. It routes an LLM judge between answering, retrieval and abstention, and it calibrates thresholds with Clopper–Pearson intervals so that the false discovery rate stays below $\alpha$. On NQ-Open with Qwen3-8B, coverage rises from 7% to 82% at $\alpha=0.20$. That work is validated only on balanced, objective factual question answering. We replace the controlled quantity with per-class miss rate. We replace retrieval with a cheap fine-tuned encoder as the first stage. We move the setting to skewed, hierarchical labels, where controlling false discoveries is close to vacuous.
OffensEval 2023 motivates the gap. It shows zero-shot LLMs trailing on OLID: 0.829 Macro-F1 for the top historical system on OffensEval 2019 against 0.793 for Flan-T5 and 0.715 for Llama-2-7B. On Arabic the gap widens to 0.902 against 0.530. The paper also names the imbalance in OLID Levels B and C as a central obstacle. It tests only one-word zero-shot prompts and proposes no remedy. We do not try to make the LLM win on F1. We ask whether it helps in the regime where the encoder is uncertain.
Dialectal Toxicity Detection is the closest work that uses LLM judges for toxicity. It finds that LLMs are self-consistent across language varieties (83.1–91.0%) but agree with humans poorly (57.2% for GPT). That study uses flat binary labels on largely synthetic dialect data. It constrains us: raw LLM verdicts on toxicity should not be trusted without calibration against human labels.
The Silent Judge shows that judges shift their verdicts on cues they never mention: a Cue Acknowledgment Rate of 0% and recency shifts of up to +30% for GPT-4o. We take this as a reason to calibrate on scores and never on the judge's written rationale.
Three shared-task systems define the scorers we must beat: the domain-pretrained RoBERTa of UHH-LT, the hierarchical multi-task model of NTU_NLP, and the back-translation augmentation of AlexU.
Few-Shot Learning with Class Imbalance separates skew inside the demonstration set from skew in the background data. That distinction shapes how we build the LLM judge's prompt: we use class-balanced demonstrations as a controlled variable.
Research Program
The system has two stages. In the first stage, a fine-tuned encoder scores every item for each node of the OLID hierarchy. The nodes are offensive or not (Level A), targeted or untargeted (Level B), and target individual, group or other (Level C). In the second stage, items whose encoder score falls in an uncertainty band are re-scored by an open-weight LLM judge. We use Qwen2.5 and Llama-3.1-class models so that label-token probabilities are available. The judge sees a hierarchical rubric prompt with class-balanced demonstrations. For black-box models, the score is the frequency of the label across sampled verdicts.
The protected event for each rare leaf class $k$ is defined end to end, from the root to the leaf. A true group-targeted insult counts as caught if the system outputs that label or defers it to a human. A miss at Level A therefore counts against the Level C guarantee. The band edges and the final thresholds are chosen jointly on calibration data. We solve
$$\min_{\theta}\ \Pr[\text{defer}] \quad \text{s.t.}\quad \mathrm{UCB}_{1-\delta}\big(\widehat{\mathrm{miss}}_k(\theta)\big)\le \beta \ \ \text{for every rare class } k,$$
where $\theta$ collects all thresholds, $\widehat{\mathrm{miss}}_k$ is the empirical miss rate on calibration items of true class $k$, and $\mathrm{UCB}$ is the Clopper–Pearson upper bound. Because many thresholds are searched, the constraint is enforced with a union bound over the searched grid, following the joint-calibration approach of Judge, Retrieve, or Abstain.
Calibration data comes from cross-fitting on the OLID training split. The encoder is trained on some folds and calibrated on held-out folds. This matters because rare classes have few examples, and the width of the Clopper–Pearson bound is set by the number of minority calibration examples. We will report how the achievable $\beta$ depends on the count of available calibration examples for each class. We consider that dependence itself a result: it tells practitioners how many labeled rare examples a guarantee costs.
Baselines
The naive reference is to send everything to humans. It has a deferral rate of 1 and zero misses. A second trivial point is an all-benign classifier, which achieves low false discovery and catastrophic recall.
The strongest competitor is the fine-tuned encoder alone, with class-conditional thresholds and deferral. We build it from domain-pretrained RoBERTa-large following UHH-LT, whose ensemble won OffensEval 2020 English Sub-task A with 92.04% Macro-F1 (91.96% for a single model). We also test a scorer built on the hierarchical model of NTU_NLP, which reports 0.7024 Macro-F1 on OffensEval 2020 Sub-task B and 0.6577 on Sub-task C, against 0.6703 and 0.6355 for plain BERT. This is the most dangerous baseline because it receives the same guarantee machinery at a fraction of the inference cost. Both encoders are trained on identical folds with the same hyperparameter search budget as our first stage. They are calibrated on the same held-out folds at the same $\beta$ and $\delta$. The cascade beats this baseline only if its deferral rate is lower at matched, verified per-class miss rates, with a paired bootstrap interval that excludes zero on the test sets.
Two more baselines complete the comparison. The first is the LLM judge alone under the same class-conditional calibration. This tests whether the LLM is useful only when paired with the encoder. The second is an FDR-controlled system built as in Judge, Retrieve, or Abstain with $\alpha$ matched. We expect it to meet its FDR target while violating our miss-rate target.
Two ablations isolate the key ingredients. In one, calibration pools all classes instead of conditioning on each class. This isolates the role of class conditioning. In the other, the second stage is a second fine-tuned encoder with a different seed or architecture instead of the LLM. This isolates whether the gain comes from the LLM or merely from ensembling two models.
Why not just…?
Why not just fine-tune the encoder and tune its threshold for recall? This is the real threat, and the proposal is built so that this outcome is a possible result. Class-conditional calibration applies to any scorer, so the encoder-only baseline receives the same guarantee. The only open question is deferral cost. OffensEval 2023 gives reason to expect the LLM to be the weaker scorer overall. Our hypothesis is limited to the uncertainty band, where encoders may be relying on surface features. If encoder-only deferral matches the cascade, we conclude that LLM judges are not worth their cost for imbalanced moderation, even as a second opinion.
Why not just correct the LLM's label priors or balance its few-shot examples? Prior correction and balanced demonstrations can raise macro-F1. Few-Shot Learning with Class Imbalance suggests that skew inside the demonstration set is what hurts. But these fixes change point accuracy and provide no bound on miss rate. We include them as variants of the LLM scorer inside the calibrated cascade, which measures whether they reduce deferral. They are components of the system, not alternatives to the guarantee.
Why not just improve the encoder with back-translation or hierarchical multi-task learning? These methods improve the scorer and do not compete with the guarantee. Back-translation raised Sub-task B validation Macro-F1 from 0.5747 to 0.6259 in AlexU. We run the cascade on top of the strongest augmented encoder, so any LLM benefit is measured against an improved first stage.
Why not just use the existing false-discovery guarantee? It bounds errors among accepted verdicts, and that quantity depends on how common each class is. The label-shift experiment below tests whether its guarantee breaks and ours holds when prevalence moves.
Proposed Experiments
The main benchmarks are OLID Levels A–C, with evaluation on both the OffensEval 2019 and 2020 English test sets. The target rare classes are group-targeted and other-targeted insults. We sweep $\beta$ over $\{0.05, 0.10, 0.20\}$ at $\delta=0.1$. The stress test is the DravidianLangTech code-mixed six-class data (Kannada, Malayalam, Tamil). There, character n-gram TF-IDF classifiers beat MuRIL on two of three languages, with weighted F1 of 0.64 on Kannada and 0.71 on Tamil in IRNLP_DAIICT, and LLMs are expected to be weakest. We therefore also include a TF-IDF scorer as a first stage there.
The metrics are the empirical per-class miss rate against $\beta$, the deferral fraction, and LLM calls per 1,000 items. Macro-F1 on accepted items is reported as secondary.
The first decisive check is validity. The empirical miss rate on each rare class must stay at or below $\beta$ across splits and random seeds.
The second decisive check concerns label shift. We calibrate at OLID prevalence, then resample the test sets so that offensive prevalence ranges from roughly one-fifth of its natural value to five times it. The prediction is that our per-class miss rate stays flat while the FDR-controlled baseline's realized recall moves with prevalence.
The third decisive check is the efficiency comparison. We compare cascade and encoder-only deferral at matched guarantees, and we use the second-encoder ablation to rule out a plain ensembling effect. If the cascade's advantage disappears once the second-encoder control is applied, the claim that the LLM contributes something fails.
References
- Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
- OffensEval 2023: Offensive language identification in the age of Large Language Models
- Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- UHH-LT at SemEval-2020 Task 12: Fine-Tuning of Pre-Trained Transformer Networks for Offensive Language Detection
- NTU_NLP at SemEval-2020 Task 12: Identifying Offensive Tweets Using Hierarchical Multi-Task Learning Approach
- AlexU-BackTranslation-TL at SemEval-2020 Task 12: Improving Offensive Language Detection Using Data Augmentation and Transfer Learning
- Few-Shot Learning with Class Imbalance
- IRNLP_DAIICT@DravidianLangTech-EACL2021:Offensive Language identification in Dravidian Languages using TF-IDF Char N-grams and MuRIL