Observer-Aligned Adversarial Training for Self-Reported Emotion

When a person writes a post and later rates how angry they felt, that rating often disagrees with what outside readers infer from the same text. Readers are systematically muted. They rate emotions as more moderate than the author reports, and they lean on obvious emotion words. Emotion classifiers are usually trained on reader labels, so they inherit this muting. The obvious fix is to train on the authors' own ratings, but such datasets are small, and small datasets invite regularization. The standard regularizer for text encoders is adversarial training: during training, the model is shown slightly perturbed versions of each input and is penalized if its prediction changes. We argue this is a trap. Adversarial training teaches a model to ignore small, low-margin differences between inputs, and those are exactly the idiosyncratic cues that separate an intense private feeling from a moderate public reading. Applied naively, adversarial training should push self-report models toward the reader's muted view.

We propose a targeted alternative. We learn from reader labels which directions in the input representation a typical observer is sensitive to, such as explicit sentiment words and stereotyped phrasing. We then apply adversarial perturbations only along those directions, so the model becomes robust to observer-salient surface cues while staying sensitive to everything else. In short, the reader's label is not used as a training target. It is used to decide which features the self-report model should be robust to.

Core Claim

We make two claims, and both are falsifiable on existing dual-perspective corpora. First, standard embedding-space adversarial training worsens the "dampening effect" when the target is a self-reported label. This holds even when it improves average error, because it smooths away the cues that predict where author and reader disagree. Second, confining perturbations to the observer-sensitive subspace keeps the regularization benefit without this damage. It measurably improves prediction of the author–reader gap, which no output-level recalibration can manufacture.

Grounding in Prior Work

The motivation comes from work on self-reported ground truth. WRIME collected 17,000 Japanese posts with both writer and reader intensity ratings for eight emotions. It found writer–reader agreement (quadratic weighted κ 0.515) below reader–reader agreement (0.547–0.585). BERT's error rose from 0.519 MAE on reader labels to 0.553 on writer labels. Variation in the Expression and Annotation of Emotions documents the same pattern in conversation: third-party raters are more neutral than participants. Understanding Annotator Perception added crowdworker labels to the 1,400 author-annotated PEACE-Reviews. It showed that training on third-person labels drops pleasantness F1 from 0.74 to 0.56–0.58 and emotion accuracy by 5–11%. Correspondence was worst for accountability and future expectancy. Can Third Parties Read Our Emotions? found that even strong LLMs reach only 0.40 macro-F1 against authors' own labels, because they also behave as observers. These papers establish the gap. None of them asks how training objectives interact with it; they rely on plain fine-tuning or multi-task learning.

The method comes from adversarial training for text. FreeLB perturbs word embeddings over several ascent steps and accumulates parameter gradients at each step. Contrastive Adversarial Training pairs one-step perturbations with a contrastive loss and reports matching full-data performance with half the data. Both treat the label as unambiguous and use a perturbation ball that is the same in every direction. We reuse FreeLB's inner loop unchanged except for the shape of the perturbation set. Both papers show that adversarial training improves clean accuracy in NLP. That result is the main reason practitioners would reach for it on small self-report corpora, and it is the result our first claim challenges for subjective targets.

Decoupling Adversarial Training for Fair NLP and domain-adversarial sentiment work (Adversarial Training Based Multi-Source Unsupervised Domain Adaptation for Sentiment Analysis) use a discriminator to remove an attribute from representations. Our design differs in kind. We do not try to make first- and third-person representations indistinguishable, which would discard the large signal the two share. We shape the local geometry the model is robust to.

Method

Let $e$ be the token embeddings of a text, and let $y_{\text{self}}$ be the author's rating. First we fit an observer model $o$ on reader labels. For each emotion dimension $j$ we compute its embedding gradient $\nabla_e o_j(e)$. Let $P_{\text{obs}}(e)$ be the projection onto the span of these gradients; with eight emotions this is at most eight directions per token. The self-report model $f_\theta$ is trained with

$$\min_\theta \; \mathbb{E}\Big[\max_{\|\delta\|\le\epsilon,\; \delta = P_{\text{obs}}(e)\,\delta} L\big(f_\theta(e+\delta),\, y_{\text{self}}\big)\Big],$$

where $\epsilon$ bounds the perturbation size. The inner maximization is FreeLB's projected ascent, with each step also projected onto the observer subspace. This turns the reader data into a statement about which input changes should not move the author's rating.

Two controls use the identical loop. The isotropic control is standard FreeLB. The observer-orthogonal control projects onto the complement, $I - P_{\text{obs}}$. Our hypothesis predicts a specific ordering on the gap metric: observer-aligned should come first, then ERM, and the other two below it, with observer-orthogonal worst. The distinguishing prediction is that the isotropic control falls below plain fine-tuning (ERM) even when its average MAE is slightly better.

Where no dual labels exist, the observer is any classifier trained on third-party labels. For the empathy data this is the frozen Ekman emotion adapter that EMOTIONSTACK already uses. For the social-media data it is a GoEmotions-trained classifier.

Evaluation Design

We measure three things beyond standard error.

The first is gap correlation: the Pearson correlation between the model's predicted author-minus-observer difference and the true author-minus-reader difference on held-out items. The observer prediction comes from the fixed observer model. This is the headline metric because it asks whether the model knows where the author departs from the crowd.

The second is dampening slope: the regression slope of predicted on true self-rating, restricted to items the author rated at the scale extremes.

The third is keyword-flip robustness. We build test variants by inserting or deleting explicit emotion words through synonym-style substitution in the manner of Alshahrani et al., restricted to cases where the author's rating is known to be unaffected, such as appended neutral clauses or hedges. We then measure how far predictions move.

The data are WRIME, PEACE-Reviews with its crowdworker annotations, the Empathy and Distress corpus of Buechel et al. and its WASSA 2022 successor, and the 729-post first-party set of Li et al. WRIME has writer labels for all posts, so it lets us simulate scarcity. We subsample writer labels to 500, 1,500 and 5,000 posts while the observer model uses all reader labels, splitting by writer so test authors are unseen.

Baselines

The naive reference is plain fine-tuning of the same encoder (BERT or RoBERTa, matching the cited setups) on self-reports alone. The strongest reported numbers are 0.553 writer-label MAE for BERT on WRIME (WRIME), 0.74 pleasantness F1 on PEACE-Reviews with first-person training (Understanding Annotator Perception), and a mean Pearson r of .424 for the CNN on Empathy and Distress (Buechel et al.). We rerun it on our splits rather than copying numbers.

Isotropic FreeLB and CAT are the strongest competitors on the regularization question. They are the established ways to make encoders robust and sample-efficient. For them, no reported result found on Lacuna for affective or self-reported datasets. They receive the same number of ascent steps and an identical grid over $\epsilon$ and step size as our method. All adversarial variants share one compute budget, measured in forward–backward passes.

On the empathy task the strongest existing system is EMOTIONSTACK from CAISA at WASSA 2022, which reports an average Pearson r of 0.523. We reproduce it and also apply observer-aligned perturbation on top of it, since the two are compatible.

On the social-media data the reference is LLM prompting at 0.40 macro-F1 from Can Third Parties Read Our Emotions?.

Two further baselines use the same reader labels we use, so any gain cannot be credited to extra data. The first is multi-task learning with reader labels as an auxiliary head, following the design in Understanding Annotator Perception. The second is post-hoc variance matching, which linearly rescales predictions so their spread matches the training self-ratings.

The isotropic and orthogonal controls are the ablations that isolate the key ingredient, the observer-defined subspace. A random subspace of equal dimension controls for dimension alone.

Beating the field means three things. The observer-aligned model must significantly exceed every baseline on gap correlation across authors (paired bootstrap over held-out writers). It must match or improve MAE, F1, or r. It must shift less than the isotropic control on keyword flips.

Why not just…?

Why not just use FreeLB or CAT as they are? Their clean-accuracy gains (FreeLB; CAT) come from objective tasks, where smoothing away low-margin cues is harmless. Our first claim is that on self-report targets this smoothing removes the signal that separates author from reader. The isotropic control tests this directly. If isotropic training matches the observer-aligned variant on gap correlation, the subspace idea is unnecessary and we will report that.

Why not just rescale the outputs to undo dampening? This is the most serious threat to the dampening-slope result. A linear rescale can restore the spread of predictions on extreme items without any change to training. It cannot, however, change which items the model ranks as departing from the observer, because rescaling preserves the ordering of predictions. Gap correlation is therefore the metric that separates the two. If rescaling recovers most of the slope improvement, we will say so and rest the claim on gap correlation and keyword-flip robustness alone.

Why not just add reader labels as an auxiliary task? Multi-task training uses the same extra information. Yet Understanding Annotator Perception reports that multi-task models built on third-person annotations still trail first-person models by 6–11%. Sharing a representation with the observer plausibly pulls the self-report head toward the observer's view. Our method instead uses the observer only to define what to be robust to. The matched multi-task baseline settles which use of reader labels is better.

Why not just use adapters or an LLM? Adapters address overfitting, not which cues get smoothed. The CAISA authors report that EMOTIONSTACK struggles most with personal distress, the most private construct. We test adapters combined with our perturbation rather than as rivals. LLMs are themselves observers, reaching 0.40 F1 against authors (Can Third Parties Read Our Emotions?), so they serve as a reference point, not a fix.

Decisive Checks

The first check is whether isotropic adversarial training lowers gap correlation and dampening slope relative to plain fine-tuning on WRIME at every scarcity level. If it does not, the tension that motivates this proposal is not real, and the paper becomes a negative result about that tension. The second check is whether the ordering observer-aligned > ERM > isotropic > orthogonal holds on WRIME and replicates on PEACE-Reviews. PEACE-Reviews is the harder test because its crowd labels are sparser and cover appraisals such as accountability, where observers are least accurate. The third check is transfer: whether a borrowed observer, the Ekman adapter or a GoEmotions classifier, still yields gains on the empathy data and on Li et al.'s posts. That would show the method does not require costly dual annotation. A positive result on all three would show that, for subjective targets, robustness is only useful when it is defined relative to what observers see.