Forget the Editors: Identity Removal as a Defense Against Temporal Drift in Front-Matter Detection

Scholarly indexes and the text corpora built from them are full of pages that are not research. These include journal mastheads listing editorial boards, conference front matter naming organizing committees, and memorial tributes with portraits and life histories. Indexes have to filter these pages out, or they create phantom co-authors and pollute training text. The same pages are also among the most person-dense content in any scholarly archive.

This note argues that the two problems are linked. A classifier that learns to recognize a masthead page partly by memorizing who sits on the board, which society logo appears, or whose face is in the tribute photo leaks personal information. It also breaks when the board turns over, the logo is redesigned, or it meets a volume from a different decade. We propose to strip identity-carrying directions out of the page representation and test one claim: that a single operation both reduces what the model knows about individuals and makes it more robust to the passage of time.

The mechanism comes from video. Privacy-Aware Video Anomaly Detection through Orthogonal Subspace Projection projected deep video features away from a "nuisance subspace" built from detected faces. Because the model could no longer memorize specific actors, anomaly detection improved while face re-identification collapsed. We carry this idea to document page images. We pick the strength of the removal using held-out future years, so that time itself becomes the yardstick for how much identity to remove.

Core Claim

For non-research document detection, three kinds of signal behave differently over time. The identity content of a page (personal names, affiliations, emails, faces, society logos) changes from volume to volume. The role content (words like "Associate Editor" and "Program Chair," roster-style multi-column layouts, the absence of abstract and reference sections) is comparatively stable. Typography drifts slowly in between.

If this holds, a page classifier trained on one era and deployed on the next fails partly because it has latched onto identity. Removing identity from its features should then shrink the gap between in-era and future-era accuracy, while also driving identity probes toward chance. The novelty is not the projection layer itself. It is treating identity removal as a temporal-generalization tool for document classification, together with a falsifiable prediction: the baseline's error should grow with roster turnover between training and test volumes, and the projected model's error should not.

Why This Intersection Is Open

Each pair of directions has been studied, but in ways that miss this point.

Document privacy work established that identity leaks through visual channels. Privacy-Aware Document Visual Question Answering showed that invoice models still recovered blurred-and-redacted provider names about 3.5% of the time, and inferred organizations from layout, typography, and logos. De-Identification of Sensitive Personal Data in Datasets Derived from IIT-CDIP found that commercial personal-data detectors miss up to 30% of some types on scans. That paper also proposed realistic pseudonymization. Both papers motivate our concern and supply baselines. However, both study contemporary commercial forms and never ask how leakage interacts with time.

Temporal drift work shows that models decay under historical and fast-moving change, but only on text transcriptions. Part-of-Speech Tagging for Historical English conditioned feature embeddings on publication year. Mitigating Temporal-Drift retrained named-entity taggers on bursty new n-grams. It found that drift concentrates on new person and organization names, with trending tweets carrying 92% more person entities. We read that finding as direct motivation: entities are the fastest-drifting layer of language. Neither paper looks at page images, and the burstiness score is poorly suited to mastheads, which recur on a fixed schedule without any surge in surface frequency.

Non-research documents themselves appear in the index as raw artifacts. Examples include IEEE Transactions on Cybernetics publication information, VLDB Front Matter, ACL Frontmatter, and tributes such as Remembering Don Bryant (1950–2024). No paper in the file treats their personal-data density as a privacy hazard or their recurrence as a drift benchmark.

Research Program

We will build a page-level benchmark of scholarly page images labeled as research or non-research. Non-research pages get one of five subtypes from the file's taxonomy: masthead, front matter or committee list, event listing or call, tribute, and tutorial or position abstract. The sources are long-running journal and proceedings series whose volumes span many years. Labels are assigned per page rather than per indexed item, because historical scans often bind front matter to the first article.

Splits are strictly chronological. We train on volumes up to year $T$, tune on $T+1$ to $T+2$, and test on later years. A second, reversed split trains on recent volumes and tests on older scans, which brings in typographic and optical-character-recognition drift. For each series we record roster turnover: the fraction of names on test-era mastheads that are absent from training-era mastheads. This turnover measure is the main explanatory variable.

The model is a standard page-image backbone. We use the Document Image Transformer that both privacy papers above rely on, and the ConvNeXt-based DocXClassifier. We insert a projection layer after the backbone's pooled features. Identity regions are located automatically. Person and organization spans come from a named-entity tagger run on recognized text, faces from a face detector, and logos from a logo detector. Every region the tagger misses is a region the method must cope with anyway.

Features pooled over these regions across a training batch are stacked, and a QR decomposition gives an orthonormal basis $Q_k$ for their top-$k$ directions. The page feature $z$ is replaced by

$$\tilde{z} = (I - Q_k Q_k^{\top})\, z,$$

which removes every component of $z$ that lies in the identity span. Following the video work, we use QR rather than a matrix inverse for numerical stability, and we keep the nuisance basis as a running estimate.

The video paper flags sensitivity to the rank $k$ as an open problem: too few directions leave identity traces, and too many erase useful signal. We turn this into a selection rule. We sweep $k$ and choose the smallest value that maximizes accuracy on the held-out future-year validation split. If the core claim is true, this rule should also land near the point where identity probes reach chance, without the probe ever being consulted.

Leakage is measured three ways, all on frozen features. The first is a linear probe trained to predict whether a given person's name appears on a page. The second is a retrieval probe asking whether a page embedding retrieves other pages that feature the same person. The third is a membership-inference attack in the style of the document question-answering work, asking whether a specific volume was in training.

Baselines

The naive reference is the unmodified backbone fine-tuned on the training era with standard augmentation. Its in-era versus future-era accuracy gap defines the problem. As a sanity anchor, DocXClassifier reports 94.17% top-1 accuracy on RVL-CDIP (DocXClassifier), so the backbone is competent at layout-driven classification. RVL-CDIP is not temporally split, and On Evaluation of Document Classification with RVL-CDIP estimates that at least 32% of its test pages share a near-duplicate or template with training. We therefore treat such numbers as inflated by exactly the kind of template memorization we are testing for.

The strongest existing competitor is realistic pseudonymization from De-Identification of Sensitive Personal Data in Datasets Derived from IIT-CDIP. That method replaces detected personal text with synthetic names rendered with scan-like degradation. The paper reports pseudonymized pages are 61.6% closer to originals in CLIP space than white-box redactions, and that classifier confidence is preserved. We run it as training-time augmentation, resampling fake names every epoch so no real or fake name can be memorized. For classification of non-research scholarly pages under temporal shift, no reported result was found on Lacuna, so we will measure it ourselves.

The second competitor is box redaction, which blacks out detected names, faces, and logos at both training and test time. The third is temporal-metadata conditioning in the spirit of FEMA-attribute, which feeds the publication year as an input embedding; on Early Modern English it reached 77.92% part-of-speech accuracy versus a 74.15% unadapted baseline (Part-of-Speech Tagging for Historical English). We also include a rule baseline built on proper-noun-to-verb ratio and the absence of abstract and reference sections, the structural signature proposed in the file.

All methods share the same backbone, training pages, number of epochs, and hyperparameter budget of twenty trials, tuned on the same future-year validation split. Two ablations isolate the key ingredient. In the first, the nuisance basis is built from random page regions of matched area rather than identity regions; if this helps equally, the benefit is generic regularization rather than identity removal. In the second, $k$ is chosen on an in-era validation split instead of the future-era one.

Beating the competitors means two things together. The projected model must show a smaller future-era accuracy drop than pseudonymization augmentation, with confidence intervals from five seeds that exclude zero difference. It must also show name-probe and membership-inference accuracy within a few points of chance, which neither the plain backbone nor redaction is expected to reach.

Why not just…?

Why not just redact the names before classifying? Redaction depends on a detector. The IIT-CDIP audit found commercial detectors miss up to 30% of some personal-data types and fail on 7 of 11 categories on scans, and our reversed split adds older fonts that degrade text recognition further. Redaction also does nothing about identity carried by logos and layout: the document question-answering work recovered provider names after both visual blurring and text redaction. Box redaction is in the baseline set. We report leakage separately on pages where the detector missed names, which is exactly where we expect redaction to fail and projection to hold.

Why not just pseudonymize the training data, randomizing names each epoch? This is the real threat. It directly prevents name memorization and may close most of the temporal gap on its own. Our expected advantages are narrower. Projection also covers faces and logos that pseudonymization cannot easily replace. It does not alter test-time pages. And it suppresses identity directions even on regions the detector did not mark, because the learned basis generalizes beyond the marked boxes. If pseudonymization matches projection on both the future-era gap and the probes, we will report that the cheaper data-side method suffices. The benchmark and the turnover analysis would remain the contribution.

Why not just retrain on recent data or condition on the year? Periodic retraining needs fresh labels for every era and does not help the reversed, older-scan split. Year conditioning lets the model learn era-specific shortcuts rather than ignore identity. The trend-score approach of Mitigating Temporal-Drift depends on frequency surges that recurring mastheads lack. The year-conditioning baseline and an upper-bound model retrained with labels from the test era quantify how much of the gap these approaches recover without addressing identity.

Proposed Experiments

The first decisive check is the turnover regression. Per series, we regress the future-era error of each model on roster turnover. The claim predicts a clearly positive slope for the plain backbone and a near-flat slope for the projected model. If both slopes are flat, identity memorization is not what drives drift on these pages, and the claim fails.

The second check is a counterfactual swap. We replace every name on test pages with names from a different series and era, and measure how often each model's prediction flips. Flip rate is a direct readout of reliance on identity, and it should fall under projection.

The third check is the $k$ curve. We plot future-era accuracy and name-probe accuracy against $k$. The claim holds if the accuracy peak occurs at or past the rank where the probe reaches chance. If accuracy peaks well before that point, removing enough identity to protect privacy costs utility, and the two goals conflict rather than align.

The fourth check asks whether the result generalizes beyond scholarly pages. We apply the same projection to RVL-CDIP resumes and letters, using the duplicate-free test subset implied by the audit of On Evaluation of Document Classification with RVL-CDIP. We test whether the gain appears where template memorization is known to inflate scores. The face-probe drop from over 70% to about 10% reported for video gives a reference point for how strong identity suppression can be. We do not expect document pages to match it exactly, because names are spread across text rather than concentrated in faces.