Safety datasets for chatbots ask many raters several narrow questions per conversation, such as whether it is harmful, biased, or misinformative, plus an overall "unsafe?" verdict. Policy decisions then depend on combinations of these answers: unsafe for a particular demographic group, or both biased and misinformative. Collecting every question from every rater is what makes such datasets expensive. DICES needed about 120 ratings per conversation on its 350-conversation subset. We propose collecting only cheap partial votes, where each rater answers one narrow question, and computing guaranteed lower and upper bounds on any combined judgment. Human ratings would then go only to items whose bounds still straddle the decision threshold.
The bounds come from an old formalism. Fuzzy description logics are logics where a concept like "unsafe" applies to an item to a degree between 0 and 1 rather than true or false. They combine degrees with fixed operators: min/max (Gödel) and the clipped-sum Łukasiewicz operators. Take the fraction of raters who say "yes" to each narrow question. If we do not know how those yes-votes overlap across raters, the tightest possible bounds on the fraction answering yes to "A or B" or "A and B" are given exactly by these two operator families. This is a standard fact from probability, the Fréchet bounds. It means a fuzzy reasoner running over vote fractions is a sound tool for budgeting human labor, and not just a modeling choice.
Core Claim
Let $p_i$ be the fraction of a rater population answering "yes" to criterion $i$ for one item. Suppose each rater's overall verdict is the disjunction of their per-criterion answers, meaning a rater calls the item unsafe if they flag any criterion. Then the population fraction $p_\cup$ calling it unsafe, and the fraction $p_\cap$ flagging all of $m$ criteria, satisfy
$$\max_i p_i \le p_\cup \le \min\Big(1, \sum_i p_i\Big), \qquad \max\Big(0, \sum_i p_i - (m-1)\Big) \le p_\cap \le \min_i p_i .$$
The lower bound on $p_\cup$ is the Gödel disjunction and the upper bound is the Łukasiewicz disjunction; for $p_\cap$ the roles reverse. Neither bound requires knowing how raters' answers correlate.
Our claim has three parts. First, these bounds, widened by finite-sample error on each $p_i$, let a reasoner settle most threshold assertions without joint ratings. A threshold assertion is a statement such as "the unsafe-degree of item $a$ for group $g$ is at least 0.5", which is exactly the $(C(a)\ge n)$ form of A Fuzzy Description Logic. Second, the items left unsettled are a small, identifiable set. Third, spending human ratings only on that set reaches full-panel decision accuracy with far fewer questions answered.
Two things would be new. The membership degrees are measured empirically from rater votes rather than assumed by a designer. And the reasoner decides who gets asked which question.
Positioning
The fuzzy-logic work supplies the machinery but leaves the numbers ungrounded. Yen introduced graded concept boundaries into terminological languages. Straccia's fuzzy $\mathcal{ALC}$ gave a sound and complete calculus and showed that computing the maximal truth degree is no harder than crisp reasoning. Steels framed descriptions as soft constraints that narrow possibilities as evidence accumulates, which is the behavior we want as votes arrive. All of these treat degrees as given inputs. We supply them from rater counts.
Ferrara et al. argued that alignment scores are "degrees of truth, not probabilities," and found that finding conflicting axiom sets did not scale to large ontologies. Our setting addresses both points. The degrees here are probabilities, and fuzzy operators are the correct sound combination when their joint distribution is unknown. Our per-item knowledge bases are also tiny (tens of criteria, a few groups), so reasoning cost is negligible.
The disagreement literature motivates the target. Fornaciari et al. showed that training on vote distributions regularizes models, but they treat each label in isolation and work offline. Lu et al. showed that LLM judges stay self-consistent (Cohen's kappa above 0.75) on items where humans split 3:2, and fall to roughly 45% accuracy on low-agreement non-offensive items. This rules out using LLM confidence to decide where humans are needed. DICES itself names the open problem of how to act when groups split evenly, which is precisely the case where our bounds straddle the threshold.
The human-in-the-loop work sets the bar. Annotator-Centric Active Learning (ACAL) chooses which annotator labels each item, but it asks the single overall question. Active Evaluation cut human comparisons with dueling bandits, and showed that weak proxy signals inflate human workload. Our proxy is a deductive bound rather than a learned score, so it cannot be confidently wrong in that way. It can only be uninformatively wide.
Research Program
Measuring how well the disjunction axiom holds. On DICES, we check per rater whether the overall verdict equals the OR of that rater's sub-criterion answers. On MHS, which annotates hate speech, toxicity and dehumanization, we test candidate axioms such as "dehumanizing implies toxic." For each axiom we report the population-level violation degree, the amount by which observed $p_\cup$ leaves its interval. Where violations are systematic, the axiom is weakened to a graded implication rather than dropped.
Building the propagation engine. The engine takes per-criterion, per-group vote counts. It converts each count into a confidence interval on $p_i$, using a Wilson interval, a standard binomial confidence interval. It then propagates these intervals through the axioms using the Łukasiewicz and Gödel operators to bound any compound query. This includes group-specific queries, which come for free by conditioning counts on rater demographics.
Choosing who answers what. The acquisition rule picks the next pair of (criterion question, rater group). It chooses the pair whose answer most shrinks the interval of a threshold assertion that is still undecided, weighted by how far the threshold sits inside the interval. When marginal votes cannot shrink an interval further, the bound gap reflects unknown correlation. Only then does the system request a joint rating, in which one rater answers all criteria, at its higher cost.
Optional learned prior. A soft-label model in the style of Fornaciari predicts $p_i$ for unrated items. It initializes the intervals, so that cold-start items are not maximally uncertain.
Baselines
Naive reference: uniform allocation. Every item receives the same number of full joint ratings, chosen to match each method's question budget. Questions answered, not raters recruited, is the shared cost unit, so that methods asking cheap marginal questions are not unfairly advantaged. Majority vote over the collected ratings gives the decision.
Strongest existing method: ACAL. We use the van der Meer et al. code and its best annotator-selection strategy, run on the same DICES and MHS splits. Its rater pool is the same simulated pool, sampled from held-out real ratings, and its question budget is identical. On DICES, ACAL reached parity with passive learning while reducing the annotation budget by 30–38% (van der Meer et al.); for MHS dehumanization the reduction exceeded 60%. Those numbers measure model training parity, not decision accuracy on compound queries. We therefore rerun ACAL under our metric and do not quote them as directly comparable. Beating it means reaching the same decision accuracy as the full DICES-350 panel, including worst-group accuracy, at a smaller question budget than ACAL needs. A gap of at least 15 percentage points of budget would count as decisive.
Uncertainty-sampling active learning. A standard active-learning baseline queries the overall question on the items with the highest model entropy. This tests whether "who" and "which question" matter beyond "which item."
Probabilistic joint model. A multi-label model with a learned correlation structure among criteria produces point estimates of compound fractions. It gets the same budget and tuning effort as our engine. This is the most serious competitor and is discussed below.
Ablations. Three ablations isolate the key ingredient. The first uses only the Gödel bounds (min/max, equivalent to assuming fully correlated raters). The second replaces the intervals with an independence-product point estimate. The third replaces the bound-based acquisition with random allocation of the same marginal questions.
Why not just…?
Why not just ask the overall question directly with ACAL? If the only downstream query were "unsafe overall, for everyone," this would likely match us, and our experiments may show exactly that. The case for our method rests on queries that were never asked. These include group-specific verdicts, conjunctions of criteria a new policy cares about, and a threshold changed after collection. We hold out a family of such compound queries and score every method on them. ACAL's labels cannot answer an unasked conjunction, while bounds over marginal votes can. If bounds answer few held-out queries decisively, the advantage disappears and we will report that.
Why not just learn the correlation between criteria and estimate compound fractions directly? This is a real threat. If inter-criterion correlations are stable, a learned joint model gives tighter point estimates than worst-case bounds. Our bet is that correlations shift across rater populations and item types, and that a confidently wrong point estimate costs more than a wide interval that triggers a joint rating. We test this directly. We fit correlations on US raters in DICES-990 and evaluate on India raters, and repeat across DICES conversation topics. We measure coverage and decision error under shift. A hybrid that uses learned correlations inside the fuzzy bounds is a natural fallback if the joint model wins in-distribution.
Why not just let an LLM judge fill in the missing ratings? Lu et al. show that LLM judges hold high self-consistency exactly where humans split, with accuracy near 60% on weak-agreement items. So their confidence cannot tell us where humans are needed. We include GPT-class judges as a baseline acquisition signal: route to humans when judge self-consistency is low. We expect it to miss the split items that DICES identifies as the hard cases.
Proposed Experiments
All experiments simulate rating collection by sampling without replacement from the real held-out ratings in DICES-350, DICES-990 and MHS. Ground truth is the full-panel vote fraction for every compound query.
Primary curve. The primary result is decision accuracy on threshold assertions as a function of questions answered, plotted for each method. It is reported overall and as worst-group accuracy across demographic groups, with the held-out compound queries reported separately.
Secondary metrics. We also report interval coverage, meaning how often the true full-panel fraction lies inside the bound, and the median interval width. As a check that the learned prior helps, we report total variation distance between the soft-label model's predicted $p_i$ and the full-panel values.
First decisive check: coverage. Coverage must stay near the nominal confidence level wherever the disjunction axiom holds per rater. Coverage loss should concentrate on items with measured axiom violations. If coverage fails elsewhere, the propagation is wrong.
Second decisive check: the kill criterion. We measure what fraction of DICES threshold assertions are settled by marginal votes alone. If $\sum_i p_i$ routinely exceeds 1 while $\max_i p_i$ sits below the threshold, the intervals are too wide to save anything, and the method fails regardless of baselines. We will report this fraction before any comparison.
Third decisive check: correlation shift. The cross-population experiment determines whether worst-case bounds beat a learned joint model. If the joint model matches us under shift, our contribution narrows to the measurement of the axioms and the coverage guarantee.
The MD-Agreement corpus has only single-label five-rater votes. It serves solely to calibrate how often split items are flagged as undecided compared with the LLM-judge routing baseline.
References
- DICES Dataset: Diversity in Conversational AI Evaluation for Safety
- A Fuzzy Description Logic
- Generalizing Term Subsumption Languages to Fuzzy Logic
- Descriptions as Constraints in Object-Oriented Representation
- Resolution of Conflicts Among Ontology Mappings: a Fuzzy Approach
- Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning
- Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement
- Annotator-Centric Active Learning for Subjective NLP Tasks
- Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons