Word-overlap and fluency scores do not measure empathy, so the field has turned to learned judges: language models fine-tuned or prompted to rate how empathetic a reply is. These judges are now used to rank systems and, increasingly, as reward signals for training them. Nobody has checked whether they can be fooled. A judge that rewards the sight of emotion words, or that drops its score after a harmless synonym swap, will be exploited by any model optimized against it. The empathy evaluation gap then reappears one level up, inside the evaluator.
This note proposes to measure that failure and to repair it cheaply. Standard adversarial training is the usual fix for a fragile model: retrain it on worst-case perturbed inputs. For a 7-billion-parameter judge working on discrete text, that is expensive, because every training step requires a combinatorial search for adversarial edits. We instead freeze the judge and train only a few learned input vectors. These vectors are pushed toward two goals at once. The judge's score should stay the same under edits that do not change empathy, and it should move under minimal edits that do. The second goal is what separates this from ordinary robustness training.
Core Claim
A small set of trained "robustness tokens" prepended to a frozen empathy judge can do two things together. It can cut the judge's vulnerability to score-inflation and score-deflation attacks, and it can keep or improve the judge's agreement with human ratings. It should do this at a small fraction of the compute that full adversarial fine-tuning needs. The ingredient that matters is a two-sided objective. Ordinary robustness work only asks a model to ignore perturbations. An empathy judge must also detect small edits that really change the meaning, such as appending "but honestly, you should get over it" to a warm reply, which turns support into dismissal. We expect an invariance-only defense to make the judge more robust and, as a side effect, less sensitive in exactly these cases.
Where This Sits in Prior Work
Three papers motivate the problem. The judges exist because the old metrics failed. Counseling-Style Reflection Generation shows that a reflection generator with ROUGE-L of only 0.126 was rated more reflection-like by humans (1.91 on a 3-point scale) than real counselors (1.85). Assessing Empathy in LLMs with Physician-Patient Interactions finds a negative correlation between perplexity and empathy. The judges are also already weak on clean data. Helping Customers In Distress reports that GPT-4.1 judges agree with human experts 76–92% of the time on objective criteria but only about 60% on empathy and frustration.
Two papers supply the judges we will attack. CharacterBench trains CharacterJudge, a Qwen2-7B model fine-tuned on human role-play ratings, and reports a 42% improvement in Pearson correlation with human judges over general-purpose evaluators. It also uses its benchmark for preference optimization (DPO, a method that trains a model directly on preferred-versus-rejected reply pairs), which makes the judge a de facto reward signal. Does GPT-3 Generate Empathetic Dialogues? defines DIFF-EPITOME, which scores three empathy mechanisms: exploration (asking clarifying questions), interpretation, and emotional reaction. None of these papers tests its judge under adversarial input.
Two papers constrain the design. A Critical Reflection on Empathy and NLP finds that 19 of 48 surveyed papers measure empathy as a single emotional-mirroring label. This predicts a specific vulnerability: judges trained on such labels may reward inserted sad words. It also tells us to collect human labels on grounded mechanisms rather than an ungrounded 1–5 scale. Dynamic Attention defends transformers by masking the tokens that receive the most attention, on the theory that attacks "hijack" attention. In empathy text, the words that draw outlier attention are often the real cues ("devastated", "panic"). We treat this as a hypothesis to test, not something to assume.
Two papers supply the method. Robustness Tokens freezes a vision transformer and trains 10 prepended vectors for about 200 steps. That restores DINOv2-G accuracy under white-box attack from 0.3% to 63.1% with clean accuracy almost unchanged (85.2% vs 85.6%). The authors suggest extending the idea to language models, but it has never been tried on text or on a scoring model. Towards Improving Adversarial Training of NLP Models (A2T) supplies the cheap attack generator. It ranks words by embedding gradients, checks meaning with DistilBERT, and regenerates adversarial examples on 20% of the data each epoch, which makes it nearly 3x faster than TextFooler-based generation. We use A2T both to make training perturbations and as the strongest full-retraining baseline.
The closest work at this intersection is EmpDG. It also pairs "adversarial" with "empathy", but in a different sense: it uses an emotional discriminator as a GAN loss to make a generator more empathetic. It does not consider attacks on a scorer, robustness, or compute cost. Our proposal reverses the setting. The discriminator-like model, the judge, is what we defend, and the threat is being optimized against.
Research Program
The first stage is an audit that builds the attack suite. We take CharacterJudge, a DIFF-EPITOME mechanism classifier, and a prompted large-model judge, and attack each in three ways.
Inflation attacks start from cold or dismissive replies and search for small edits that raise the judge's score. These include A2T-style word substitutions and a gradient-free "mirroring insertion" attack that adds emotion words or stock phrases such as "I'm so sorry you're going through this".
Deflation attacks start from high-quality reflections taken from the motivational interviewing data and the PsyQA counseling answers, and search for meaning-preserving substitutions that lower the score.
Flip pairs are minimal edits written by hand and by a language model that genuinely remove empathy (invalidation, premature advice, subject changes) or genuinely add it.
For every perturbed item we collect human ratings on the three EPITOME mechanisms, so that "attack success" means the judge's score moved while the human rating did not. For flip pairs, success means the judge's score failed to move when the human rating did. We will collect roughly two thousand rated pairs, which is the main human-labeling cost of the project.
The second stage is the defense. Let $J_\phi$ be the frozen judge with parameters $\phi$, and let $T$ be $k$ trainable token embeddings prepended to its input, with $k$ around 10–20. Write $J_\phi(T; x)$ for the score of input $x$ with the tokens attached and $J_\phi(x)$ for the score without them. We minimize
$$ \mathcal{L}(T) = \mathbb{E}\big[(J_\phi(T; x) - h(x))^2\big] + \lambda_1\,\mathbb{E}\big[(J_\phi(T; \tilde{x}) - J_\phi(T; x))^2\big] + \lambda_2\,\mathbb{E}\big[\max(0,\, m - s\,(J_\phi(T; x') - J_\phi(T; x)))\big]. $$
Each term has one job. The first keeps the judge anchored to human ratings $h(x)$ on clean data. In the second, $\tilde{x}$ is an empathy-preserving perturbation of $x$; this term asks for invariance, which is the original Robustness Tokens objective. In the third, $x'$ is a flip-pair edit of $x$ whose human rating moves in direction $s \in \{-1, +1\}$, and $m$ is a margin; this term requires the score to move by at least $m$ in the correct direction. A2T-style perturbations are regenerated against the token-augmented judge on a subsample each round, so the inner search is cheap and only the $k$ vectors receive gradients.
The third stage tests whether hardening matters in downstream use. We run best-of-$n$ selection and DPO on EmpatheticDialogues generators, using each judge variant as the reward, and then have humans rate the outputs. A fragile judge should show reward hacking: judge scores climb while human empathy ratings stay flat or fall, for example through formulaic emotion insertion. A hardened judge should keep these two curves closer together.
Compute is reported throughout as GPU-hours of hardening and as cost per scored item at inference.
Baselines
The naive reference point is the unmodified judge: CharacterJudge as released, the DIFF-EPITOME classifier, and the prompted large-model judge. For agreement with humans the best figure we can anchor to is CharacterBench's 42% relative Pearson improvement over general-purpose judges, plus the roughly 60% human agreement on empathy that Helping Customers In Distress reports for GPT-4.1. No reported result on the adversarial robustness of any empathy judge was found on Lacuna, so the audit establishes that number. Beating the naive judge means lowering attack success on the human-verified suite without reducing clean Pearson correlation.
The strongest existing method is full adversarial fine-tuning of the judge with A2T perturbations (Towards Improving Adversarial Training of NLP Models). It is the established way to make text models robust and it updates every parameter, so it has strictly more capacity than our tokens. For a fair comparison it gets the same perturbation generator, the same human-labeled flip pairs added as training targets, and the same hyperparameter search budget. We report it both at its natural compute and at compute matched to the token method. Beating it means equal robustness at a fraction of the GPU-hours, or better flip-pair sensitivity at equal cost. If full A2T fine-tuning with flip pairs wins clearly at matched compute, the token method has no case beyond low cost, and we will report that.
The zero-training competitors are two inference-time methods. The first is the "Adversarial Possibility" prompt from Improving Adversarial Robustness in VLMs with Architecture and Prompt Design, adapted to tell the judge that the reply may be manipulated to look empathetic. In its original vision-language setting this prompt raised LLaVA-7B accuracy under Auto-PGD on COCO from 12.54% to 91.46%. The second is Dynamic Attention applied to the judge. Both cost nothing to train and are scored on the same suite.
Two ablations isolate the key ingredient. The first trains the tokens with the invariance term only ($\lambda_2 = 0$), which reproduces Robustness Tokens directly. We predict this improves attack resistance but hurts flip-pair accuracy, which would show that the two-sided term is necessary. The second trains the same tokens on clean human ratings with no perturbations ($\lambda_1 = \lambda_2 = 0$). This is ordinary prompt tuning, and it separates the benefit of adversarial data from the benefit of simply adding trainable parameters.
Why not just…?
Why not just fully adversarially fine-tune the judge with A2T? This is the real threat. A2T is already about 3x faster than standard generation and it updates every parameter. The paper also reports that its context-aware variant sometimes hurt clean accuracy when substitutions changed meaning. For an empathy judge, that is the case where invariance and sensitivity collide. The matched-compute comparison and the flip-pair metric settle the question directly. If full fine-tuning matches us on both at equal cost, the contribution reduces to the audit and the two-sided objective, which can then be applied to any training method.
Why not just tell the judge it may be attacked? The prompt result in the vision-language paper is striking, but the same paper notes that an attacker who knows the prompt can optimize through it. A prompt also cannot teach a judge which edits change empathy. We test it under adaptive attacks that include the prompt text. If it holds up, it becomes the recommended zero-cost default and the token method must justify its cost by the flip-pair gain alone.
Why not just apply Dynamic Attention? It needs no training and was effective against TextFooler and PWWS on classification. It masks the most-attended tokens, though, and in empathy text those are often the emotional cues the judge needs. We measure clean correlation and flip-pair accuracy with and without it. A drop on emotionally loaded inputs would confirm this conflict.
Why not just use a bigger judge or an ensemble? The 7B-to-13B scaling result in Improving Adversarial Robustness in VLMs showed negligible robustness gains, and GPT-4.1 already sits near 60% agreement on empathy. The prompted large-model judge in our audit tests this directly. If the mirroring-insertion attack inflates its scores as well, scale is not the fix.
Decisive Checks
The project succeeds if three things hold together. First, on the human-verified inflation and deflation suite, the token-hardened CharacterJudge shows clearly lower attack success than the released judge under adaptive white-box attacks that also optimize against the tokens. Without that adaptive condition, the result would inherit the "secret key" weakness that Robustness Tokens leaves open. Second, it keeps clean Pearson correlation with human EPITOME ratings on CharacterBench and EmpatheticDialogues within noise of the original. Third, it beats the invariance-only ablation on flip-pair accuracy.
The downstream check is the most convincing evidence. In best-of-$n$ and DPO runs on EmpatheticDialogues, the gap between rising judge scores and human-rated empathy should be smaller under the hardened judge than under the original. A full cost table in GPU-hours should show where the token method sits relative to A2T fine-tuning. If the original judge turns out not to be fragile, meaning the audit finds low attack success, that negative result is still worth publishing, because it would justify the field's current reliance on learned judges.
References
- Counseling-Style Reflection Generation Using Generative Pretrained Transformers with Augmented Context
- Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions
- Helping Customers In Distress: An LLM-Powered Agent That Converses, Probes, and Routes
- CharacterBench: Benchmarking Character Customization of Large Language Models
- Does GPT-3 Generate Empathetic Dialogues? A Novel In-Context Example Selection Method and Automatic Evaluation Metric for Empathetic Dialogue Generation
- A Critical Reflection and Forward Perspective on Empathy and Natural Language Processing
- Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Towards Improving Adversarial Training of NLP Models
- EmpDG: Multi-resolution Interactive Empathetic Dialogue Generation
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design