Hierarchical multi-label text classification assigns each document several labels drawn from a taxonomy, such as "computer science" and "machine learning". Recent progress in this area comes largely from contrastive training. Contrastive learning trains an encoder to place documents with shared labels close together in embedding space and documents with different labels far apart. Every number behind that progress comes from old public corpora: RCV1-v2 news, Web of Science abstracts, NYTimes, book blurbs, and arXiv abstracts. Modern encoders may have read much of this text during pretraining. This note proposes a controlled experiment to measure what that exposure does to the scores, and in particular whether it helps the fine-grained, deep labels more than the coarse ones. We deliberately inject known test documents into an encoder's pretraining and compare those documents with matched documents the encoder never saw. We also measure the same models on fresh arXiv abstracts posted after the encoder was trained.
The question matters because the field's main open trade-off sits exactly at depth. Jump To Hyperspace compares contrastive losses for hierarchical multi-label classification. It finds that losses weighted by label overlap win on overall micro-F1 and exact-match ratio (the fraction of documents whose full label set is predicted correctly). Losses that model the hierarchy's asymmetry win on rare leaf labels. Leaf decisions often hinge on a few distinctive words. Those words are also what an encoder memorizes when it has seen a document before. If familiarity with test text inflates leaf accuracy, then both the size and the direction of that trade-off could change on genuinely new documents. The calibration guarantees built on top of these classifiers would also be overstated.
Core Claim
We hypothesize that encoder exposure to test documents produces a depth-dependent gap in classification quality. The gap is near zero at the coarse, top levels of the taxonomy and grows toward the rare leaves. Its size depends on the training objective, so loss rankings reported on static splits partly measure how much each loss exploits memorized text.
We further hypothesize a downstream effect on uncertainty estimates. Hierarchical conformal prediction produces prediction sets made of taxonomy nodes, calibrated on held-out data, that are meant to contain the true label a fixed fraction of the time. When the calibration documents are familiar to the encoder, their confidence scores are too optimistic. The resulting sets should then cover the true label less often than promised on fresh documents, and the shortfall should concentrate at deep nodes.
What is new is the controlled, depth-resolved measurement, not a new architecture. Prior contamination work either studies flat question-answering tasks or detects that something changed. DyVal regenerates reasoning problems from graphs and shows models collapse on unseen items. CANARY audits hidden states for poisoning. Neither measures how memorization interacts with hierarchical depth, or with the choice of representation-learning loss.
Research Program
Controlled injection. The first component is a causal contamination experiment. For each benchmark (RCV1-v2, WOS-46985, NYTimes, Blurb Genre Collection, AAPD), we split the official test set into two halves at random, stratified by leaf label so both halves have the same label distribution. We then continue the encoder's masked-language-model pretraining on one half. Each injected document is mixed into a generic text stream at one, three, or ten exposures.
We run two versions of the injection:
- Text only. The injected documents appear without labels, mimicking a crawled article.
- Label-leak. The label strings are appended to each document, mimicking an arXiv listing page that prints the category next to the abstract.
The un-injected half is the matched control. Because the split is random and stratified, any per-depth difference between the halves is caused by exposure. We define the contamination gap at taxonomy depth $d$ as
$$\Delta_d = F1_d(\text{injected}) - F1_d(\text{held}),$$
where $F1_d$ is the micro-F1 computed only over labels at depth $d$.
Error propagation. We also measure how mistakes travel down the tree. The propagation rate is $\pi_d = P(\text{label at depth } d \text{ wrong} \mid \text{its parent correct})$. It separates errors made at a level from errors inherited from a wrong coarse decision. We report $\pi_d$ separately for injected and held documents. This shows whether memorization repairs leaf choices directly, or instead stabilizes the coarse decisions that leaves depend on.
Natural test. The second component tests whether the injected effect appears without our intervention. AAPD's labels are arXiv categories, so new abstracts posted after the encoder's pretraining cutoff can be labeled automatically from arXiv metadata under the same taxonomy, with ancestor nodes added as AAPD does. A temporal split mixes two causes: exposure, and topic drift (new papers discuss new things). We therefore compare the post-cutoff gap with the injected gap. We also add a drift control: pre-cutoff abstracts that are absent from the encoder's documented corpus, identified by exact n-gram matching where the corpus is public. We run each experiment on two encoders: BERT-base, matching prior work, and one encoder whose documented pretraining corpus includes arXiv text.
Objectives compared. The third component asks which training losses absorb memorization. Each encoder variant is fine-tuned on the official training split with the following objectives:
- Binary cross-entropy: an independent yes/no prediction per label.
- Jaccard-weighted multi-label contrastive loss: pairs of documents are pulled together in proportion to their label-set overlap, as in Jump To Hyperspace.
- Label-aware entailment-cone loss: from the same paper, which encodes the parent-child direction of the hierarchy.
- HILL: HILL, which avoids text augmentation and aligns document embeddings with a structure encoder built from the label hierarchy.
- Information-weighted overlap (our variant): a contrastive loss that tests one specific mechanism, described next.
The mechanism is this. AAPD and RCV1 add all ancestors to every label set, so Jaccard overlap is dominated by shared coarse ancestors. Two documents that share only a root and a mid-level node then count as strongly similar. Our variant replaces the plain overlap with
$$s(i,j) = \frac{\sum_{\ell \in Y_i \cap Y_j} w_\ell}{\sum_{\ell \in Y_i \cup Y_j} w_\ell}, \qquad w_\ell = -\log p(\ell \mid \text{parent}(\ell)).$$
Here $Y_i$ is document $i$'s label set, and $p(\ell \mid \text{parent}(\ell))$ is the training-set frequency of label $\ell$ among documents carrying its parent. A shared ancestor that nearly every document has contributes almost nothing. A shared rare leaf contributes a lot. The variant is a probe, not the headline. If the leaf deficit of Jaccard contrastive loss is caused by ancestor domination, this change should close it on held and post-cutoff documents. If the deficit instead reflects memorization, the change should not close it there.
Conformal readout. The fourth component tests uncertainty estimates. We take softmax scores from each trained model and build node-valued prediction sets with the hierarchical conformal methods CRSVP and CRSVP-r from Conformal Prediction in Hierarchical Classification. CRSVP returns a single internal node; CRSVP-r allows up to $r$ nodes, which makes sets smaller. We calibrate once on injected documents and once on held documents, then measure coverage and set size on fresh post-cutoff documents. We also apply the robust quantile correction (CRCP) from Split Conformal Prediction under Data Contamination to see whether a correction designed for noisy calibration sets also repairs memorization-induced optimism.
Baselines
The naive reference: trust the static split. We report every model's official-test-split numbers, as prior papers do, and treat them as what the field currently believes. For context, the original RCV1 paper reports a support vector machine at micro-F1 0.816 on topics (RCV1). If $\Delta_d$ is near zero at every depth, for every loss, at every exposure level, the naive reference is vindicated and the static numbers can be trusted. That outcome would be a useful negative result, and we state it as a real possibility.
The strongest published method: HILL. HILL reports average gains of 1.85 points micro-F1 and 3.38 points macro-F1 over vanilla BERT across WOS, RCV1-v2, and NYTimes (HILL). The file contains no per-dataset absolute numbers. It is the strongest competitor here because it was explicitly designed to preserve the fine-grained word evidence that distinguishes child classes, which is the evidence memorization would also strengthen. We run it with its released configuration and give every method the same tuning budget: the same number of hyperparameter trials, selected on the same validation split, with the same encoder checkpoint for each contamination condition.
"Beating" HILL in this study does not mean a higher static score. It means one of two findings:
- HILL's advantage over BERT on held or post-cutoff documents is significantly smaller than on injected documents, which shows part of its reported gain is exposure-dependent.
- Some other loss shows a smaller held-versus-injected gap at deep levels while matching HILL's held-document macro-F1 within noise.
The trade-off pair from Jump To Hyperspace. These are the Jaccard contrastive loss and the label-aware cone loss in Euclidean geometry. The file reports no absolute numbers for them, only that Euclidean and hyperbolic differences in micro and macro F1 are often below 0.1, which we read as a 0.1 F1 margin. They are the direct test of whether the published trade-off survives on held and post-cutoff documents.
The key ablation. The information-weighted overlap is compared with plain Jaccard overlap and with class-balanced reweighting of binary cross-entropy. This isolates whether discounting shared ancestors matters beyond simply upweighting rare labels.
Conformal baselines. These are CRSVP, CRSVP-r, and CRCP. For scale, the hierarchical conformal paper reports a CRSVP-3 set size of about 44 versus about 123 for CRSVP on PlantCLEF 2015 at 90% coverage. CRCP reports restoring 91.7% coverage with 2.18-class sets where standard conformal prediction gave 99% coverage and 5.47-class sets on CIFAR-10N. The file has no reported conformal result on any text taxonomy.
Why not just…?
Why not just use a temporal split and skip the injection? A post-cutoff split is the most realistic test, and we use it. But it confounds exposure with topic drift: new arXiv papers use new vocabulary and new subfields. A drop in leaf accuracy could therefore reflect either cause. The randomized injection removes drift entirely, because both halves come from the same time and distribution. Agreement between the injected gap and the post-cutoff gap after the drift control is what licenses calling the natural gap contamination. If the two disagree, we will report drift as the dominant factor rather than argue for memorization.
Why not just flag memorized test items by low perplexity and drop them? Memorized evaluation items do show unusually low token-level surprise, as documented by Yax et al.. This is a real threat to the need for our heavier design. Our injection provides ground truth, so we test this alternative directly: we report the AUROC of encoder pseudo-perplexity for separating injected from held documents. If perplexity filtering recovers injected items well and the filtered static scores match post-cutoff scores, practitioners should use the simple filter. Our recommendation would then shrink to "filter by perplexity", and we will say so. Our expectation is that a single exposure lowers surprise only weakly while still shifting leaf predictions, but that is an empirical question this proposal settles.
Why not just reweight rare leaves instead of changing the contrastive overlap? Class-balanced or frequency-weighted binary cross-entropy targets the same leaf deficit with no contrastive machinery. It is included as an ablation with an equal tuning budget. If it matches the information-weighted overlap on held-document leaf F1, the ancestor-domination explanation is unnecessary and we will drop the variant. The contamination findings do not depend on that variant.
Why not just run CANARY to detect contamination? CANARY detects that a fine-tuned model's hidden states have shifted relative to a clean base model (CANARY). It answers whether a model was altered, not which test items are familiar or how familiarity changes per-depth scores. It also needs an uncontaminated reference checkpoint, which does not exist for public encoders that already read arXiv. Our injection design creates such a reference only for the controlled arm.
Proposed Experiments and Decisive Checks
The primary result is the depth profile of $\Delta_d$ and $\pi_d$, for each loss, exposure level, and injection variant, with bootstrap confidence intervals over test documents and three fine-tuning seeds.
Decisive check 1: is the gap depth-dependent? The hypothesis predicts that $\Delta_d$ increases with depth under text-only injection. It fails if the gap is flat across depths, or largest at the root.
Decisive check 2: does the loss ranking survive? We test whether the published ranking of Jaccard contrastive loss over the cone loss on exact-match ratio, and the reverse ranking on leaf F1, holds on held and post-cutoff documents. A reversal, or a collapse of the margin below seed noise, would mean part of the published trade-off came from exposure.
Decisive check 3: does calibration break? Conformal sets calibrated on injected documents should fall measurably below 90% coverage on post-cutoff documents, with deep nodes losing the most. Calibrating on held documents should restore coverage. We also test whether CRCP restores coverage. The CRCP paper itself notes that subtle contamination produces under-coverage it handles poorly, which predicts that CRCP will not.
Secondary checks. We compare the label-leak and text-only injections. This measures how much of any effect needs labels to co-occur with documents, and so bounds how worried the field should be about benchmarks whose labels are printed beside the text on the web. Finally, we report the perplexity-filter AUROC described above.
References
- Jump To Hyperspace: Comparing Euclidean and Hyperbolic Loss Functions for Hierarchical Multi-Label Text Classification
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models
- HILL: Hierarchy-aware Information Lossless Contrastive Learning for Hierarchical Text Classification
- Conformal Prediction in Hierarchical Classification with Constrained Representation Complexity
- Split Conformal Prediction under Data Contamination
- RCV1: A New Benchmark Collection for Text Categorization Research
- Studying and improving reasoning in humans and machines