Short essays written by British 11-year-olds in 1969 predict how distressed those same people report being at ages 23, 33, 42 and 50. The predictions are modest, and they get worse the further out they reach. By age 42, knowing only a person's gender beat every language model submitted to the benchmark. One reason is that many word–outcome links are not stable over the life course. A childhood mention of "husband" or "school" signals good adjustment at 11 but higher distress at 23. A model fit to whatever horizons have labels learns these links, and they may not carry to a horizon it has never seen. We propose to train text models so that their features relate to distress the same way at every observed age. The bet is that such features reflect durable traits rather than age-specific accidents, and so extrapolate better to unseen, more distant ages.
The mechanism comes from adversarial domain adaptation, which trains a feature extractor to fool a second network, the discriminator, that tries to tell which "domain" an example came from. Here each outcome age is a domain. The input essay is identical across domains; only the label changes. A discriminator shown the essay's representation alone therefore has nothing to detect. We instead show it the pair (representation, outcome score) and ask it to guess which age the score was measured at. Fooling it forces the relation between features and outcome to be age-invariant. In practice, this means the extractor stops relying on any feature whose link to distress differs between the training ages.
Core Claim
The headline claim is that conditional invariance across observed horizons improves prediction at held-out, more distant horizons, compared with an equally tuned model that simply pools horizons. The relevant comparison is correlation with distress at ages 42 and 50 when only ages 11, 23 and 33 are used for training.
The second, sharper claim concerns setting. This is an unusually clean place to use adversarial alignment, because two known failure modes largely disappear when inputs are fixed across domains. The first is mode mismatch from aligning marginal feature distributions, which Conditional Adversarial Domain Adaptation diagnosed and fixed by conditioning the discriminator on predictions. The second is distortion of the regression mapping under label shift, warned about in On Target Shift in Adversarial Domain Adaptation. In our setting the marginal over inputs is already aligned, and we rank-normalize labels within each horizon. That leaves the adversary acting only on the conditional relationship.
We are explicit that the principle, not the adversary, is the main claim. A simpler linear analogue keeps only features whose correlations are stable across ages. It is a serious competitor and is built into the test.
Prior Work and Where This Sits
Several papers motivate the proposal. The CLPsych 2018 shared task, CLPsych 2018 Shared Task: Predicting Current and Future Psychological Health from Childhood Essays, defined the task on the National Child Development Study (NCDS). It reported that accuracy decays with horizon and that a gender-only baseline won at age 42.
Current and Future Psychological Health Prediction using Language and Socio-Demographics of Children found the sign flips of words like "husband" and "school". It named the problem "temporal domain adaptation" but used unadapted ridge regression on unigrams. It also showed that lexicons and personality models trained on modern social media transfer poorly to 1969 essays. We take that as a constraint on relying on modern pretrained encoders.
Two papers supply components we reuse. Team UKNLP's CNN system used convolutional encoders with LIWC (a psychological word-category dictionary) and demographics, Huber loss, and a classify-then-regress "switch" head for zero-inflated scores. It trained each horizon as an independent head, which is exactly the setup we change. UGent-IDLab showed that character n-grams capture spelling idiosyncrasies that carry signal.
Two linear-model papers constrain the design. A Psychologically Informed Approach and Top-Down Text Representation and Simple Bottom-Up Model Selection found that simple theory-driven linear models with misspelling, word-count and affect features are competitive and resist overfitting. Our method must therefore beat linear stability selection, not just a CNN.
The closest methodological neighbour is ATOP. It aligns essay representations across different essays on different prompts with a domain discriminator, and splits shared from topic-specific prompts for essay scoring. We differ in what the adversary sees. ATOP aligns input distributions across topics, whereas we align feature–label relationships across time on fixed inputs, and we predict future distress rather than present writing quality. ATOP is also a cautionary result: aggressive alignment stripped useful syntactic signal on structured essays. This is why we measure what the invariant model throws away.
The survey Adapting Language Models to Temporal Drift treats drift as a change in the inputs between eras. Our drift lives in the meaning of fixed inputs for outcomes at different life stages.
Research Program
Let $x$ be an essay and $h$ an outcome horizon. Let $\tilde y_h$ be the outcome at that horizon after rank-normalizing within horizon, which removes marginal label shift. Let $G$ be the text encoder, $P$ a shared prediction head, and $D$ a discriminator over horizons. Let $\phi(\tilde y)$ be a smooth embedding of the scalar outcome, such as radial basis features, so the CDAN-style product $G(x)\otimes\phi(\tilde y)$ works for continuous targets. Let $\lambda$ be the adversarial weight. Training solves
$$\min_{G,P}\ \max_{D}\ \sum_{h\in\mathcal{H}_{\text{train}}} \Big[\, \mathcal{L}_{\text{Huber}}\big(P(G(x)),\tilde y_h\big) \;-\; \lambda\, \mathcal{L}_{\text{CE}}\big(D(G(x)\otimes\phi(\tilde y_h)),\,h\big)\Big].$$
The training horizons $\mathcal{H}_{\text{train}}$ are ages 23 and 33, optionally with age 11 (BSAG). The objective is implemented with a gradient reversal layer, which passes features through unchanged on the forward pass and flips the sign of the discriminator's gradient on the way back.
One shared head across horizons is deliberate. It encodes the hypothesis that a single trait-like mapping underlies all ages. A per-horizon intercept and the Rios-style zero-inflation switch remain, so differences in base rates are not forced into the features. The encoder combines a character CNN, a word CNN and the hand-crafted mechanical features: misspelling rate, word count, and asterisk counts for illegible words. Demographics enter only at the head, never into $G$, so the adversary cannot quietly use or erase them.
The BSAG is teacher-rated while the Malaise Inventory is self-reported, so including age 11 mixes instruments. We will run with and without it.
Hyperparameters, including $\lambda$, are chosen using training horizons only. We use leave-one-training-horizon-out validation: train on 23, validate on 33, and the reverse. Age 42 labels are never touched until final evaluation. Without this discipline the experiment would be meaningless.
The analysis has two parts. First, we inspect which words and character patterns lose weight under invariance, testing directly whether the Guntuku sign-flip words are suppressed. Second, we measure what the model gives up at near horizons. An invariant model should lose a little at age 23 and gain at 42 and 50. If it gains everywhere, the effect is more likely generic regularization than invariance.
As a secondary, deliberately small replication, we would apply the linear version of the idea to the Danish deployment cohorts in Karstoft et al.. There we would treat the cohort × horizon cells (2.5 and 6.5 years) as environments. With a few hundred soldiers per cohort, we do not expect adversarial training to be stable there.
Baselines
The naive reference points are the training mean and a gender-only ridge model. A gender-only baseline won outright at age 42 according to the shared task overview, though no reported number for that baseline was found on Lacuna. We also fit gender plus parental social class, which Guntuku et al. found the more durable predictor over decades. Beating this baseline at 42 and 50 is the minimum bar for any claim that language helps at long range.
The strongest reported number is from Çöltekin et al.: an average disattenuated Pearson correlation of 0.319 across ages 23, 33 and 42, as reported in the shared task overview. Disattenuated correlation corrects Pearson correlation for the unreliability of the psychometric scales. The file does not describe that system well enough to reimplement it, so we report it as a reference number only.
The strongest reproducible system is UKNLPA from Rios et al.: a CNN with LIWC and gender, fed to ridge regression, reported at an average of 0.3062 on Task B. We reimplement it on our encoder and splits. Two caveats apply. Its published number used age 42 labels in training, which ours will not. And the shared-task test averages include age 42, so our held-out comparison is a stricter setting than theirs.
Every learned model gets the same splits, the same encoder family, the same hand-crafted features, the same demographic inputs at the head, and the same hyperparameter search budget, counted in trials. Each model is selected by the same training-horizon-only validation rule.
The decisive ablations isolate the key ingredient. In the first, the discriminator sees only $G(x)$ without the label. On fixed inputs this should be vacuous and match the non-adversarial model, which is a check that any gain comes from conditioning on the outcome. In the second, horizon labels are shuffled before being given to the discriminator, which controls for the adversary acting as generic noise. In the third, the non-adversarial model is regularized more strongly with extra dropout and weight decay, tuned on the same budget.
Beating these baselines means a statistically reliable improvement in disattenuated correlation at age 42 over pooled training, the gender plus class baseline, and both ablations. Reliability is judged by bootstrap over test participants with paired comparison. The same direction at age 50 counts as confirmation.
Why not just…?
Why not just use gender and social class? At age 42 this may well win; the shared task says it already did. Our design treats that as the bar, not an embarrassment. The invariance claim is also a claim about why language lost there: models fit horizon-specific word links that do not carry forward. If the invariant model still fails to beat gender plus class at 42 and 50, we learn that the essay carries no durable signal beyond demographics. That would be a clean negative result the field currently lacks.
Why not just average the adult outcomes into one target, or pool horizons? This is a real threat. Averaging across ages partly cancels the age-specific components and so implicitly favours stable features. The pooled shared-head model, trained on stacked 23 and 33 labels and on their mean, is our main comparison for exactly this reason. Averaging penalizes unstable links only through noise reduction. The adversary removes them explicitly, so we predict the gap appears mainly for features with opposite-sign associations, like the "husband" example. Feature-level inspection will show whether that is what happens.
Why not just keep the features whose correlations are stable across ages, or add a penalty on how much the loss varies across ages? These are the linear and non-adversarial versions of the same principle. Given the Simchon and Gilead evidence that simple linear models are competitive here, they could match the adversarial model. We therefore run stability-selected ridge and a variance-of-risk penalty on the same CNN encoder as co-equal arms. If either matches the adversary, the contribution becomes the invariance principle and the cheaper method. We would report it that way rather than defend the gradient reversal layer.
Why not just fine-tune a pretrained transformer? Guntuku et al. found modern-text models transfer poorly to these essays. ATOP reports that full BERT fine-tuning trailed adversarial prompt-tuning across topics. Neither result concerns horizon extrapolation, however, so we include a fine-tuned pretrained encoder trained with and without the adversary. This tests whether invariance helps regardless of encoder.
Proposed Experiments
All main experiments use the NCDS essays: 9,217 for training and 1,000 for testing under the shared-task split. Malaise Inventory outcomes at ages 23, 33, 42 and 50 provide labels, with BSAG at age 11 as an optional extra environment. Age 50 outcomes were withheld in the shared task, so using them requires access to the full cohort records. We assume that access, and fall back to age 42 alone if it is unavailable.
The primary protocol trains on ages 23 and 33 and evaluates on 42 and 50. A secondary protocol trains on 23, 33 and 42 and evaluates on 50, for comparability with shared-task practice. Metrics are disattenuated Pearson correlation, mean absolute error, and AUC for identifying the top decile of distress. The zero-inflated scales make the clinically relevant minority easy to wash out, and correlation alone hides that.
Four checks decide the outcome.
The first is the held-out-horizon comparison: does conditional invariance beat pooled training, gender plus class, and the shuffled-horizon and label-free-discriminator controls at age 42?
The second is the near-versus-far trade-off. A small loss at 23 alongside a gain at 42 and 50 supports the invariance account. A uniform gain points to regularization.
The third is the feature audit. Words whose association flips sign between 11 and 23 should lose weight. The mechanical writing features, which Iserman et al. found predictive at several ages, should survive. If invariance instead strips the misspelling and illegibility signal, it is repeating ATOP's failure of removing useful structure.
The fourth is the head-to-head among the adversary, stability selection and the variance penalty. It determines which instantiation of the idea earns its complexity.
The Danish replication uses a linear stability method with leave-one-cohort-out evaluation. Its reference point is the out-of-cohort AUC of 0.79 that Karstoft et al. report for transfer from the 2009 to the 2013 cohort. Because those cohorts share a near-identical mission, we expect little headroom. We treat that experiment as a test of whether invariance does no harm, not as evidence for a gain.
References
- Conditional Adversarial Domain Adaptation
- On Target Shift in Adversarial Domain Adaptation
- CLPsych 2018 Shared Task: Predicting Current and Future Psychological Health from Childhood Essays
- Current and Future Psychological Health Prediction using Language and Socio-Demographics of Children for the CLPysch 2018 Shared Task
- Predicting Psychological Health from Childhood Essays with Convolutional Neural Networks for the CLPsych 2018 Shared Task (Team UKNLP)
- Predicting Psychological Health from Childhood Essays. The UGent-IDLab CLPsych 2018 Shared Task System
- A Psychologically Informed Approach to CLPsych Shared Task 2018
- An Approach to the CLPsych 2018 Shared Task Using Top-Down Text Representation and Simple Bottom-Up Model Selection
- Adversarial Topic-Aware Prompt-Tuning for Cross-Topic Automated Essay Scoring
- Applicability of an Automated Model and Parameter Selection in the Prediction of Screening-Level PTSD in Danish Soldiers Following Deployment: Development Study of Transferable Predictive Models Using Automated Machine Learning