Heavy-Tailed Tree Fields for Clustered Annotation Errors

Benchmark labels are wrong more often than leaderboards admit, and the errors are not scattered at random. In the TACRED relation-extraction benchmark, re-annotation found 23.9% of labels incorrect. The errors pile up in recognizable pockets: sentences with the same entity-type pair, the same misleading cue word, or the same annotator shortcut, especially a default to no_relation (Re-TACRED; TACRED Revisited). Most noise-robust training methods treat each example's chance of being mislabeled as independent of every other example's. That throws away the most useful signal we have: if a sentence's close neighbors were mislabeled, it probably was too.

This note proposes to learn a hidden "error propensity" score for every training example. The score is smoothed along a sparse tree that links similar examples, and the smoothing uses heavy-tailed jumps rather than Gaussian ones. A heavy-tailed distribution puts real probability on rare, large values. Here that means the score stays nearly constant inside a pocket of similar sentences but can change abruptly at the pocket's edge. Error clusters can then be sharply bounded instead of bleeding into clean neighbors. In short, we model annotation noise as a piecewise-constant field over a similarity tree, infer it with tree-structured variational inference, and use it to down-weight and re-attribute suspect labels while training a relation extractor.

Core Claim

Our hypothesis is that annotation noise in sentence-level benchmarks is both clustered and sharply bounded. If so, a model that captures both properties will do two things better than models that capture neither or only one. It will detect which training labels are wrong, and it will train a more accurate extractor on the noisy labels.

Concretely, a Student-t-increment tree field over example-level error propensities should outperform three alternatives. The first is independent per-example noise variables, which capture no clustering. The second is a Gaussian tree field, which captures clustering but over-smooths across cluster boundaries. The third is non-generative neighbor heuristics.

TACRED makes this testable. Re-TACRED re-annotated the training split as well as the test split, so we know which training labels were actually wrong. Few label-noise papers can check their inferred noise against such per-example ground truth.

The Model

Each training sentence $i$ has an observed label $y_i$, an unobserved true label $y^*_i$ predicted by a neural classifier $p_\theta(y^*_i \mid x_i)$, and a scalar error propensity $\eta_i$.

We assume the observed label is either the true label or a corruption of it. The probability of corruption is $\sigma(\eta_i)$, where $\sigma$ is the logistic function. A corrupted label is drawn from a small learned transition matrix $C_{g(i)}$, one matrix per entity-type pair $g(i)$. These matrices are free to put most of their mass on no_relation, which reflects the negative bias documented by Alt et al..

Error propensities live on a spanning tree $\mathcal{T}$ over training sentences. Each child's score equals its parent's score plus a jump drawn from a Student-t distribution. We write the Student-t jump as a Gaussian with a random per-edge scale:

$$\eta_{c} = \eta_{\text{pa}(c)} + \sqrt{w_{c}}\,\epsilon_{c},\qquad \epsilon_{c}\sim\mathcal{N}(0,s^2),\quad w_{c}\sim\text{Inv-Gamma}(\nu/2,\nu/2).$$

In this equation, $\text{pa}(c)$ is the parent of node $c$ in the tree, $s$ is a global jump scale, and $\nu$ is the degrees of freedom (smaller values give heavier tails).

The scale-mixture form solves a technical problem. The KL divergence (the term in the variational objective that measures how far the approximate posterior is from the prior) has no closed form between Student-t distributions; Sparse Variational Student-t Processes had to use sampling or a Jensen bound to handle it. Writing the jumps as Gaussians with random scales sidesteps this. Once the edge scales are fixed, the field is a Gaussian tree. We approximate the posterior over $\eta$ with a tree-structured Gaussian, sampled through the sparse lower-triangular transform $\eta = \mu + L\epsilon$ that TreeVI introduced. That transform has linear cost and runs in parallel on a GPU. For each edge scale we use an independent inverse-gamma posterior, so both KL terms are closed form.

The objective is the evidence lower bound, the standard variational training objective:

$$\mathcal{L}=\sum_i \mathbb{E}_{q}\Big[\log \sum_{k} p_\theta(y^*_i{=}k\mid x_i)\,p(y_i\mid y^*_i{=}k,\eta_i)\Big]-\mathrm{KL}\big(q(\eta,w)\,\|\,p(\eta,w)\big).$$

The sum over the true label $k$ runs over 42 classes, so it is computed exactly. When $\sigma(\eta_i)\to 0$ for every example, the objective reduces to cross-entropy. This mirrors the reduction that the ECCD segmentation paper proves for its spatial model.

This design also defuses a tension the research file raises. Heavy-tailed losses can simply ignore badly mismatched examples, while TIER-A argues that memorization shows up as collapsing prediction entropy. We do not put heavy tails on the classifier's loss. They go only on the noise field. A suspect label is explained by the noise model rather than silently dropped, and the classifier is never rewarded for becoming confident on it.

The tree is built rather than learned end-to-end. Learning the structure with an acyclicity constraint, as TreeVI does, is costly at about 68k training sentences. Instead we compute a minimum spanning tree over frozen encoder embeddings of each sentence and its entity markers, restricted to edges within the same entity-type pair. We rebuild the tree once per epoch from the current encoder.

Relation to Prior Work

ECCD (A Bayesian Approach to Segmentation with Noisy Labels via Spatially Correlated Distributions) is the closest prior work and supplies the template of a correlated latent field that drives label flips. Its field is Gaussian, and its covariance has a special banded form (a Kac-Murdock-Szegö matrix) that only makes sense on regular pixel grids. We move the idea to an irregular graph over sentences and replace the Gaussian increments with heavy-tailed ones.

TreeVI and HoT-VI supply the inference machinery for correlations across examples. Both assume Gaussian perturbations, and neither addresses label noise. HoT-VI also needs a pre-specified backbone, a constraint we accept by building the tree from embeddings.

The Student-t VAE and SVTP show why heavy tails help: Gaussian likelihoods fail under outliers and tight clusters. Both apply heavy tails to continuous observations under unstructured or feature-space approximations. We apply them to the increments of a latent structured field over discrete labels.

On the label-noise side, two methods treat examples as conditionally independent. PTD models example-dependent noise as a linear mixture of transition matrices fit by point estimation. TIER-A regularizes entropy across an ensemble of 2–3 models. Our proposal differs from both by coupling examples explicitly and by producing a posterior over which labels are wrong.

Research Program

The work proceeds in three stages, each of which can stop the project.

The first is a structure audit that needs no new model. Using the per-sentence Re-TACRED corrections on the TACRED training split, we measure how strongly the "was this label wrong" indicator agrees between neighbors on the embedding tree. We compare that agreement to random trees and to trees built only within entity-type pairs. We also measure how corruption changes along tree paths, to check whether it is blocky (long runs of the same value with sharp transitions) or smooth. If errors do not cluster on the tree, the structured component is unjustified and we stop. If they cluster but change smoothly, the Gaussian tree should suffice and the heavy-tail claim fails.

The second stage implements the model on a single RoBERTa-class or SpanBERT encoder. We start from the SpanBERT codebase and add the error field, the transition matrices, and the tree sampler. We expect a training-time overhead similar to TreeVI's reported 2–3×.

The third stage is controlled synthetic noise. We start from Re-TACRED training labels, treated as approximately clean, and inject corruption clusters on the tree. We vary two dials independently: cluster size and boundary sharpness, where boundaries range from hard edges to gradual Gaussian falloff. This maps out where heavy-tailed jumps should help and where they should not.

Baselines

The naive reference is standard cross-entropy fine-tuning of the same encoder on TACRED training labels. As context for how much a clean test set shifts scores, TACRED Revisited reports that average F1 across 49 models rose from 62.1 to 70.1 when they were evaluated on TACREV. Any denoising gain must be read against that scale.

The upper reference is the same encoder trained on Re-TACRED-corrected training labels. This shows how much of the clean-label gap each method closes.

The strongest existing competitor is TIER-A. It jointly trains 2–3 copies of the network with temperature scaling, a penalty on low prediction entropy, and a soft vote at test time. It is the only method in the file evaluated on this exact benchmark and noise. It reports F1 gains of 3.1% to 3.8% on the TACRED relabeled test set across ALBERT, BERT and LUKE, and about 74.5% F1 when 50% of labels are flipped. The file gives no absolute TIER-A F1 for a matched clean-test setup, so we will rerun it ourselves rather than quote a target.

We will run TIER-A two ways on the same encoder, data, and hyperparameter search budget. The first is its native 2–3 model ensemble. The second is a compute-matched version, in which our single model receives the same number of training passes in total, either through extra seeds or through a seed ensemble of equal size. Beating TIER-A means higher micro-F1 on both TACREV and Re-TACRED test sets. The margin must exceed the standard deviation across five seeds, and it must hold in the compute-matched comparison.

PTD is the representative transition-matrix method. We adapt it by using encoder features in place of image parts. Its anchor-point step is fragile when 80% of examples carry the majority label, so we report it with and without anchors taken from the Re-TACRED-validated subset. Its image results (71.67% on Clothing1M versus 68.88% for cross-entropy) show it is strong where its assumptions hold.

A neighbor-consistency baseline captures structure without a generative model. It down-weights a training label in proportion to how much its k nearest neighbors in the same embedding space disagree with it.

Two ablations isolate the key ingredient. The first keeps everything identical but makes the jumps Gaussian. This is effectively ECCD's correlated field moved onto our tree, and it isolates the heavy tails. The second keeps Student-t propensities but removes the tree, giving every example an independent heavy-tailed $\eta_i$. This isolates the structure. A random-tree control on top of that separates the value of similarity structure from the extra parameters the tree adds.

Why not just…?

Why not just relabel the data? Re-TACRED did exactly that, which is why we can evaluate at all. Human relabeling covered one benchmark at real cost and could only be released as a patch because of licensing (Re-TACRED), and TACREV relabeled only about 5,000 evaluation sentences (TACRED Revisited). The model is also complementary to relabeling. We will measure how many relabeling decisions, prioritized by the posterior corruption probability, are needed to recover half of the clean-label F1 gap, compared with random or confidence-based prioritization.

Why not just smooth labels over nearest neighbors? This is the most serious threat, because a spanning tree in embedding space is close to a k-nearest-neighbor graph. The difference is that neighbor smoothing cannot tell "this region is hard to classify" apart from "this region was systematically mislabeled toward no_relation." Our transition matrices and our propensity field separate these two explanations. If the neighbor baseline matches our corruption-detection precision (measured as area under the precision-recall curve against Re-TACRED corrections) and our downstream F1, we will report that the generative model adds nothing beyond structure.

Why not just use an independent heavy-tailed noise variable per example? The no-tree ablation is exactly this. The evidence from TreeVI that modeling correlations between examples helps comes from clustering (81.4% versus 80.5% accuracy on Fashion-MNIST constrained clustering), not label noise, so it does not settle the question here. The structure audit and the ablation do.

Why not just use a Gaussian correlated field like ECCD? If the audit shows errors change smoothly along the tree, the Gaussian tree should match ours, and we would drop the heavy-tail claim. Our prediction is specific. The gap should open as cluster boundaries become sharper in the synthetic sweep, and it should be largest on the real TACRED noise only if the audit finds blocky errors.

Why not just use TIER-A? It is cheap and effective, but it couples examples only implicitly and produces no per-example posterior over which labels are wrong. The two are not exclusive. We will also report TIER-A's ensemble on top of our objective.

Decisive Checks

The first check is corruption detection. On the TACRED training split, does the posterior mean of $\sigma(\eta_i)$ rank the truly corrupted labels (per Re-TACRED) above the clean ones better than all baselines and both ablations? We measure this with precision-recall and ROC curves, reported separately for false no_relation errors and for errors between two positive relations.

The second check is downstream accuracy. Is micro-F1 on TACREV and Re-TACRED test higher than TIER-A at matched compute? We also check whether the result reproduces the ranking changes Re-TACRED reported, such as C-GCN pulling ahead of PA-LSTM only once labels are cleaned.

The third check is the synthetic boundary-sharpness sweep. It should show the Student-t tree pulling ahead of the Gaussian tree only when boundaries are sharp. If the gain is uniform across the sweep, the heavy tails are acting as a generic regularizer, not doing what we claim.

As a secondary check on transfer, we run the same model on Clothing1M against PTD's reported 71.67% accuracy, with the tree built over image embeddings.