Seed Noise or Real Ambiguity? Training Trajectories as Functions for Patronizing Language Detection

Classifiers for patronizing and condescending language, meaning text that talks down to a group while sounding kind ("these brave refugees just need our help"), are unstable. Retrain the same model with a different random seed and it flags a different set of paragraphs. Shared-task systems have mostly averaged this instability away with ensembles. That leaves an important question unanswered. When two seeds disagree about a paragraph, is the model picking up something humans also find ambiguous, or is it just luck of the draw? The question matters because the leading view in this field treats disagreement as signal: human annotators disagree on these paragraphs, and some of the best systems learn from that disagreement. Yet seed-to-seed disagreement in text classifiers is known to be largely arbitrary.

We propose to answer the question with a tool from functional data analysis, the branch of statistics that treats each observation as a whole curve instead of a list of numbers. For every paragraph we record the model's predicted probability of "patronizing" at many points during training, across many seeds. This gives each paragraph a bundle of curves over training time. We then split each bundle into two parts. The first is a shared curve that all seeds follow, which reflects what the data itself drives. The second is the seed-specific wiggle around that curve, which is initialization luck. Our bet is that the shape of the shared curve tracks human disagreement, while the seed wiggle mostly does not. If that holds, the split gives a practical training signal: soften the examples that are genuinely ambiguous, and stabilize the ones that are merely lottery-prone.

Core Claim

Per-example seed variance in fine-tuned patronizing-language classifiers is a mixture of two things: pragmatic ambiguity, and initialization noise unrelated to the text. The two can be separated. The separation becomes possible once each example's training history is treated as a smooth function of training time and decomposed into a seed-shared mean curve and seed-specific deviations. Scalar summaries such as final-probability variance, or the average and spread of confidence over epochs, mix the two because they discard curve shape. Here, "shape" means things like being learned early and then forgotten, oscillating, or being learned late but consistently.

The claim makes two testable commitments. The diagnostic commitment is that features from the shared curve predict annotator disagreement better than scalar summaries do, while seed-deviation energy predicts it poorly. The practical commitment is that using the split to set training targets lowers seed-to-seed spread on the Don't Patronize Me benchmark without lowering mean F1. It should do so more efficiently than an ensemble at matched compute.

Why this setting, and what prior work leaves open

The application is the Don't Patronize Me dataset, introduced by Don't Patronize Me!. It contains about 10,000 news paragraphs scored 0–4 by annotator consensus, with roughly 9.5% positive examples and seven fine-grained categories, one of which has only 69 training instances. It is unusually well suited to our question because it ships a graded measure of human ambiguity. The 0–4 score acts as a ground truth for "humans found this hard," and borderline paragraphs scored 2 were excluded from the SemEval-2022 Task 4 test set. Prior work on this data motivates our question in two opposite ways. DH-FBK treats annotator disagreement as informative and adds uncertainty and agreement prediction as auxiliary tasks, reporting +26.9 F1 on the multi-label subtask without ensembles. Ensemble-based entries such as SSN_NLP_MLRG and Hu et al. treat seed variance as a nuisance to average away. Neither line asks whether the model's instability lines up with the humans' instability.

The initialization-sensitivity literature constrains what we should expect. Reimers & Gurevych showed across 50,000 sequence taggers that seeds alone swing NER F1 between 89.99% and 91.00%. They also found that dev and test performance correlate weakly across seeds (ρ = 0.229), so choosing the best seed on validation is unreliable. Mehrer et al. found that class centroids stay consistent across seeds (up to r² = 0.95) while the arrangement of individual examples diverges in deep layers (0.44–0.62). This is exactly the per-example instability we target. Slack et al. showed that token attributions overlap only about 53–56% across seeds on standard text benchmarks, which warns that much of what differs between seeds is arbitrary. Together these suggest that some, but not all, of the per-example disagreement on Don't Patronize Me is noise. Our contribution is to measure that fraction directly.

The method comes from functional data analysis. Yang & Narayanan turned discrete turn-level synchrony scores into B-spline curves (piecewise-polynomial smooth curves) and used functional principal components to find shape patterns that scalar averages missed, such as oscillating gesture synchrony that tracks valence (r² = 0.719). We reuse that pipeline, but our "time" axis is training steps and our curves are a model's beliefs about an example. FuDGE supplies a caution: the number of retained components controls a bias–noise tradeoff, so we report results across basis sizes. The closest conceptual neighbor is Calibrating Deep Ensemble through Functional Variational Inference, which views seed diversity in function space over inputs in order to calibrate ensembles. We instead study per-example functions over training time, and we test them against an external measure of human ambiguity. That work does neither.

Research Program

Trajectory collection. We fine-tune RoBERTa-base, the organizers' baseline architecture, with one fixed hyperparameter setting under K = 20 seeds. Each seed varies weight initialization of the classification head, data order, and dropout. We record P(PCL) for every training and development paragraph at T ≈ 50 evenly spaced checkpoints. We repeat the process with the adapter configuration that the pre-training analysis found strongest, to test whether the decomposition holds under parameter-efficient tuning. Training RoBERTa-base on this data is cheap, so 20 seeds is affordable.

Functional decomposition. For paragraph $i$ and seed $k$, let $p_{ik}(t)$ be the smoothed probability curve over normalized training time $t \in [0,1]$, obtained by B-spline smoothing of the logit at each checkpoint. We write

$$p_{ik}(t) = \mu_i(t) + \epsilon_{ik}(t),$$

where $\mu_i(t)$ is the mean across seeds and $\epsilon_{ik}(t)$ is the seed deviation. From this we derive two per-example quantities. The first is a shape description of $\mu_i$: its scores on functional principal components fitted across all paragraphs. The second is the lottery energy $L_i = \frac{1}{K}\sum_k \int_0^1 \epsilon_{ik}(t)^2\,dt$, the average squared seed deviation over training. This is a functional ANOVA in the style of Yang & Narayanan's analysis, applied to a new axis.

Diagnosis. On development paragraphs, which are never trained on but are scored at every checkpoint, we test which quantities predict annotator disagreement. We measure disagreement in two ways: distance of the 0–4 score from the extremes, and membership in the borderline score-2 set. We also check whether high-$L_i$ paragraphs are the ones whose final labels flip between seeds. We then break the results down by the seven categories, to see whether lexically marked categories such as Metaphor behave differently from context-heavy ones such as Presupposition.

Intervention. We retrain using the decomposition. Examples with ambiguous shared-curve shapes get soft targets derived from $\mu_i$. Examples with high lottery energy but unambiguous shared curves get consistency regularization toward $\mu_i$, which acts as distillation from the seed-averaged trajectory into a single model. The deployed model is one network. It needs the K seed runs once, at analysis time, not at inference.

Baselines

The naive reference point is a single RoBERTa-base run, reported as the full distribution over 20 seeds, not the best seed, following Reimers & Gurevych. For context, Don't Patronize Me! reports 70.63% binary F1 for RoBERTa on its own split. The SemEval overview reports that top binary submissions reached F1 in the mid-60s and top multi-label systems reached macro-F1 in the high 40s. Tesla reports 0.1041 average F1 for the RoBERTa multi-label baseline. No reported result was found on Lacuna for the seed-to-seed standard deviation of any model on Don't Patronize Me, so we establish that number ourselves. We count a reduction in seed standard deviation of at least a third, at equal or higher mean F1, as success.

The strongest competitor for stability is a seed ensemble at matched compute, meaning the same 20 training runs averaged at test time. It is the strongest because it is what the leading shared-task entries actually used, and it attacks seed variance directly. We beat it if our single distilled model matches the ensemble's mean binary F1 within its confidence interval while costing 1/20 the inference. We would also count it as beating the ensemble if the ensemble preserves seed instability on exactly the ambiguous paragraphs where our model flags uncertainty.

The strongest competitor for using ambiguity is a reimplementation of DH-FBK's multi-task model with uncertainty and agreement auxiliary heads, run on the same RoBERTa-base backbone, the same split, and the same 20 seeds. A related baseline trains directly on soft labels derived from the 0–4 scores. Both use human disagreement explicitly. Our method uses only model trajectories at training time, so matching them shows that trajectories carry the same information. Exceeding them on stability, or combining with them for further gains, shows the information is complementary.

The ablation that isolates our key ingredient replaces the functional decomposition with scalar summaries: final-probability variance across seeds, and the mean and spread of confidence across checkpoints. All downstream steps stay identical. All methods receive the same hyperparameter search budget of 20 configurations on the development split and are evaluated on the SemEval test split.

Why not just…?

Why not just ensemble seeds? An ensemble reduces output variance but hides the instability instead of explaining it. Mehrer et al. show that per-example geometry still diverges underneath averaged outputs. An ensemble also cannot say which disagreements reflect real ambiguity, and that is the scientific question here. On raw F1 an ensemble is a genuine threat. If it beats our single distilled model by more than its seed confidence interval, we will report the decomposition as a diagnostic contribution only and drop the claim that it is a better training recipe.

Why not just use scalar per-example variance? This is the most serious threat. If final-probability variance or confidence spread separates ambiguity from noise as well as curve shape does, the functional machinery is unnecessary. The ablation settles this directly. We commit in advance to abandoning the functional claim if the functional features do not beat scalar summaries by at least 0.05 AUROC on predicting borderline paragraphs, with 95% bootstrap intervals that do not overlap. If that happens, the paper becomes a measurement study of how much of the seed variance on this benchmark is ambiguity versus noise. That remains a useful result.

Why not just train on the annotators' soft labels? Soft labels encode human ambiguity but do nothing about seed lottery on paragraphs humans agreed on. Our hypothesis is that these unambiguous but unstable paragraphs are a sizable part of the spread. The soft-label baseline tests this. If soft labels alone shrink seed standard deviation as much as our method, then the lottery component was not worth targeting.

Proposed Experiments and Decisive Checks

The first decisive check is the diagnostic correlation on development paragraphs. We measure the Spearman correlation between annotator disagreement and shared-curve shape scores, and separately between annotator disagreement and lottery energy $L_i$. The hypothesis predicts a clearly positive correlation for the shared curve and a weak one for $L_i$. If $L_i$ correlates as strongly as the shared curve, then seed disagreement on this task is mostly ambiguity. That result would vindicate the assumption behind DH-FBK and would itself be worth publishing.

The second check is predictive. Using shape scores alone, we predict membership in the score-2 borderline set, which is excluded from the official test set, and compare against the scalar ablation by AUROC. We repeat the comparison over B-spline knot counts from 5 to 20, to rule out a basis-choice artifact of the kind FuDGE warns about.

The third check is the intervention. We report the binary F1 mean and standard deviation over 20 retraining seeds on the SemEval test split, and multi-label macro-F1 with the 69-instance "poorer, the merrier" category reported separately, where instability should be worst. We compare against all baselines under the matched budget.

The final check tests whether the result generalizes beyond this one data source. Within Don't Patronize Me, we hold out the paragraphs from a subset of source countries entirely, as a modest distribution shift. We then test whether high-$L_i$ examples still predict which held-out paragraphs flip between seeds. Tesla's drop from above 0.80 validation F1 to 0.2445 test F1 shows that this is where methods built on this benchmark tend to fail.