Many text-labeling tasks assign each sentence a role in its document: "background" or "result" in a medical abstract, "thesis" or "supporting idea" in a student essay. Both people and models lean heavily on where a sentence sits. An abstract's first sentence is usually background and its last is usually a conclusion. A model that knows only a sentence's position already does well, and human annotators who see the full document can use the same cue. This note argues that the two ways the field reads results on these tasks, comparing models to human inter-annotator agreement (how often two trained people give the same label) and testing models on a second, independent dataset, share a hidden common factor: positional convention. Where layout is conventional, agreement looks high and models look strong. Where it is not, both fall apart, and the second dataset mostly reveals how much the first dataset's conventions were doing.
We propose to measure this directly and then correct for it. First, we split agreement and accuracy by how "positionally typical" each sentence is, and we run a re-annotation study in which the same sentences are labeled with document order shown or hidden. Second, we propose a simple model that factors its prediction into a text part and a position part, so that at transfer time the position part can be re-estimated on the new corpus's unlabeled documents. This step needs no target labels. If the shared-factor claim holds, the agreement ceiling should be reported per position, and cross-dataset gains should be read on the sentences that position does not already explain.
Core Claim
The claim has three testable parts. First, human agreement on sentence-role labels is substantially higher on sentences whose label matches what position alone would predict, and it drops when annotators cannot see document order. Second, the regime in which models match or exceed human agreement is concentrated on those positionally typical sentences. This is where the "compromise" and "consistency" mechanisms of Inter-annotator agreement is not the ceiling of machine learning performance can operate cheaply, because the position prior is a low-noise signal that humans also apply, though less consistently. Third, cross-dataset losses concentrate on positionally atypical sentences and on positions where the two corpora's conventions differ. These losses are therefore largely recoverable by re-estimating the position prior on unlabeled target text, without touching the text encoder.
How This Relates to Prior Work
Several papers motivate the proposal. Hierarchical sentence position encodings are known to matter within a single corpus. Discourse Self-Attention for Discourse Element Identification in Argumentative Student Essays encodes a sentence's index in the essay, its paragraph's index, and its index within the paragraph, and reports an 8% macro-F1 drop when these encodings are removed. Identifying the Information Structure of Scientific Abstracts finds sentence location to be a dominant feature for SVM and Naive Bayes models on cancer-risk abstracts. It also reports agreement of κ=0.84 on a coarse 4-class scheme versus κ=0.50 on an 11-class scheme. Neither paper asks whether its annotators' agreement itself rested on position, and neither tests on a second corpus. We test both.
Two other lines constrain the design. Spurious Correlations in Cross-Topic Argument Mining shows that topic words dominate what cross-topic argument models rely on, and that stripping open-class words helps distant transfer. Their target is lexical shortcuts. Ours is the structural analogue, and position may be a shortcut too rather than the clean, invariant signal that Gap 1 of our source file assumes. Towards More Generalizable and Accurate Sentence Classification in Medical Abstracts with Less Data shows that prompt learning with full abstract context transfers well between randomized-trial and observational-study abstracts. Its prompts mix text and position, so it cannot say which one transfers. Its labels also come from author-written section headings rather than independent annotators, so it says nothing about agreement.
Two papers supply methods we reuse or contrast with. Learning part-of-speech taggers with inter-annotator agreement loss weights training errors by how often humans confuse two tags and improves out-of-domain tagging. Its confusion weights are the same at every position in the sequence; we condition them on position. Should You Fine-Tune BERT for Automated Essay Scoring? finds that BERT and n-gram models land within 0.04 Quadratic Weighted Kappa of each other near the human-agreement level. We read this as consistent with our hypothesis that a cheap shared signal saturates measured performance, though essay scoring is a document-level task and we do not test it directly.
Research Program
The first component is a measurement of position-conditioned agreement. We define the positional typicality of a sentence $s$ as $\tau(s) = p_{\text{pos}}(y_s \mid g_s, r_s, l_s)$. Here $p_{\text{pos}}$ is a position-only classifier trained on the source corpus. It takes three inputs: the sentence's relative offset in the document $g_s$, its paragraph's relative offset $r_s$, and its offset within the paragraph $l_s$. The quantity $y_s$ is the reference label. We report agreement, measured as Cohen's κ or pairwise F1, and model F1 separately in typicality bins.
Existing labels are insufficient for this, because the PubMed datasets have heading-derived labels and multi-annotator releases are scarce. We therefore run a re-annotation study of roughly 300 documents per corpus with three annotators per sentence. Each document is labeled in one of two conditions: with order and paragraphing shown, or with sentences presented in shuffled order. The drop in agreement from the shown condition to the hidden condition is our estimate of how much of the "ceiling" is positional. Collecting three labels per sentence also lets us run the Richie-style test on real text: whether the model agrees with the majority of the other two annotators more often than a held-out annotator does, computed per typicality bin.
The second component is a factorized classifier,
$$p(y \mid x, \pi) \propto \frac{p_{\text{text}}(y \mid x)\, p_{\text{pos}}(y \mid \pi)}{p(y)},$$
where $x$ is the sentence with its neighboring sentences, $\pi = (g, r, l)$ is its three-level position, and $p(y)$ is the label marginal. Both experts are trained jointly. The text expert is a PubMedBERT or RoBERTa sentence encoder with context, and the position expert is a small network over $\pi$. At transfer time we keep $p_{\text{text}}$ fixed and re-estimate $p_{\text{pos}}$ on the target corpus by expectation-maximization over unlabeled target documents, treating target labels as hidden. This is label-shift correction made position-specific, and it targets exactly the layout differences that our source file lists as an uncontrolled confound in cross-dataset transfer.
We also test a position-conditioned version of Plank et al.'s confusion-weighted loss. It uses confusion matrices estimated separately per typicality bin from the re-annotation data.
Baselines
The naive reference is a position-only classifier, with no text. On each target corpus it shows how much a layout prior alone achieves and how much that prior degrades under transfer. We found no reported result on Lacuna for a position-only classifier on any of these corpora, so we compute it ourselves.
The strongest existing system on the medical abstracts is the prompt-learning approach of Towards More Generalizable and Accurate Sentence Classification in Medical Abstracts with Less Data. That paper reports cross-dataset F1 of 0.9065 when training on PubMed 20k RCT and testing on PubMed 20k OS, and 0.9203 when training on OS and testing on RCT. It is the strongest competitor because its templates already give the model the whole abstract. Any gain we attribute to explicit position handling must therefore exceed what a full-context language model gets for free. We re-run it on the same PubMedBERT backbone, data splits, and hyperparameter search budget as our model. Beating it means higher macro-F1 on positionally atypical target sentences, with overall F1 no lower, averaged over three seeds with bootstrap confidence intervals that exclude zero.
On essays, the strongest system is DiSA from Discourse Self-Attention for Discourse Element Identification in Argumentative Student Essays. It reports in-corpus macro-F1 of 0.699 on English essays and 0.657 on Chinese essays. We found no reported cross-dataset result on Lacuna for DiSA. We train it on one essay corpus and test it on another under a shared coarse label mapping, using the same encoder as our model.
For the cancer-risk abstracts, the reference is the SVM from Identifying the Information Structure of Scientific Abstracts, which reports 89% accuracy on the coarse scheme at κ=0.84.
Three ablations isolate the key ingredient. In the first, the factorized model is trained without EM re-estimation, which isolates the transfer-time prior update. In the second, position features are dropped entirely. In the third, Plank et al.'s uniform confusion-weighted loss is compared with our position-conditioned version.
Why not just…?
Why not just drop positional features so the model cannot use the shortcut? In-corpus, this costs about 8% macro-F1 according to Discourse Self-Attention, and position is genuinely informative on targets whose conventions match the source. The real question is whether dropping position does as well as re-estimating it on corpus pairs whose layouts differ, such as observational studies versus trials, or essays from different school systems. That is a direct ablation in our design. If dropping position matches the factorized model on atypical sentences and loses only on typical ones, we will report that EM adds nothing.
Why not just use a full-context pretrained model, which already sees order? The model of Towards More Generalizable and Accurate Sentence Classification in Medical Abstracts already transfers above 0.90 F1, so this alternative is a real threat. Its reliance on position is implicit, however, and cannot be updated for a new corpus without target labels. We settle the question by comparing it with our model on the atypical-sentence bin and on synthetically re-ordered target abstracts. If it matches us there, the factorization is unnecessary.
Why not just use Plank et al.'s agreement-weighted loss? It does improve out-of-domain tagging in Learning part-of-speech taggers with inter-annotator agreement loss, with chunking F1 rising from 68.49 to 68.97 on the Foster test set. Its confusion matrix is the same at every position, though, and our first hypothesis predicts that human confusion is strongly position-dependent. The comparison between the uniform and position-conditioned loss tests this. If the re-annotation study shows flat agreement across typicality bins, the position-conditioned variant has no rationale and we drop it.
Why not just label a few target documents? Few-shot target adaptation is a fair upper reference. We include a 50-document labeled-target fine-tuning condition to show how much of that gap unlabeled EM closes.
Proposed Experiments and Decisive Checks
The corpus pairs are as follows. Within medicine, we transfer between PubMed 20k RCT and PubMed 20k OS in both directions. To reach a different genre and annotation process, we map the PubMed classes onto the coarse 4-class cancer-risk scheme, using the subsumption structure reported by Guo et al. For essays, we transfer between the English and Chinese essay corpora used by DiSA, mapping both to their shared coarse roles. Re-annotation covers a sample from each corpus.
The first decisive check concerns humans. If agreement does not drop when document order is hidden, and does not differ between typicality bins, the core claim fails, and the gains we report from position re-estimation are ordinary domain adaptation.
The second check concerns models. On real three-annotator data, the model should exceed held-out-annotator agreement mainly in the high-typicality bin. A uniform advantage across bins would support Richie et al.'s view without the positional qualification.
The third check concerns transfer. Most of the cross-dataset loss should fall in the low-typicality bin and in positions where the source and target position priors differ most, measured as KL divergence between the two corpora's $p_{\text{pos}}$ distributions. The factorized model with EM should recover a measurable share of that loss relative to the full-context baseline.
A final stress test re-orders sentences in target abstracts. If performance tracks the original label of each sentence, the model is reading text. If it tracks the new position, the model is reading layout, and we can quantify how much of each baseline's reported transfer is layout.
References
- Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive set of simulations
- Discourse Self-Attention for Discourse Element Identification in Argumentative Student Essays
- Identifying the Information Structure of Scientific Abstracts: An Investigation of Three Different Schemes
- Spurious Correlations in Cross-Topic Argument Mining
- Towards More Generalizable and Accurate Sentence Classification in Medical Abstracts with Less Data
- Learning part-of-speech taggers with inter-annotator agreement loss
- Should You Fine-Tune BERT for Automated Essay Scoring?