Measuring "Happiness-by-Default" in Empathetic Speech Agents Against Human Listeners

Conversational AI that "responds with empathy" is being deployed for emotional support faster than anyone can say what an unacceptable emotional response looks like. A sharp critique of this trend, Computer says "No", argues that such systems push users toward calm and cheerfulness by default. That habit can suppress negative emotions that are legitimate and useful, such as justified anger. The argument is purely normative and has never been measured. We propose to measure it.

The test works like this. Take real two-person conversations from IEMOCAP, an acted corpus of paired speakers recorded in improvised and scripted scenes. Stop each conversation just after one speaker expresses a negative emotion. Have a speech agent produce the listener's next turn, then compare what it does with what the human listener actually did in the same moment. Two questions drive the comparison. Does the agent steer away from the speaker's feeling more often than people do? Does that tendency get worse when the distress is carried by the voice rather than the words?

We build the audit as separable stages: perception, agent response, and rule check. That lets us locate each failure. If the agent hears the distress and still smooths it over, the fault is in how it generates responses. If it never hears the distress, the fault is in perception. The stages also let us re-score stored conversations whenever governance rules change, without regenerating anything.

Core Claim

The claim is that the happiness-by-default tendency can be measured once two conditions hold. First, the reference point must be human listener behaviour at matched speaker states, not an abstract ideal. Second, the speaker's state must be read from the acoustics as well as the transcript. Under these conditions the measure should track human judgments of dismissiveness better than a judge that reads only the transcript. The gap should be largest on turns where the words are mild but the voice is aroused.

The acoustic requirement has a basis in prior work. Busso and Rahman found that acoustic features predict activation well on the VAM talk-show corpus (r = 0.8035) and valence poorly (r = 0.3245). This both motivates and constrains the design. Voice is where arousal lives, so a transcript-only audit should miss exactly the turns that are lexically neutral but vocally escalated. Valence, however, has to come mainly from the words, so the perception stage cannot be audio-only.

Research Program

The pipeline has three stages that exchange stored intermediate records. This mirrors the four-stage split that OmniEvaluator uses to separate data, inference, post-processing and metrics in general model evaluation. We reuse the design pattern only. OmniEvaluator does not evaluate affect, and its verifier judges only text.

The perception stage reads the speaker's turn and writes a record of two estimates. The first is an arousal estimate from audio. The second is a distribution over emotion categories that combines audio with lexical content. We use a distribution because the ambiguous-emotion work on large audio-language models shows that single labels throw away mixed states that annotators actually perceive. The perception stage can be swapped. Its options are oracle IEMOCAP annotations, the LFST+STEE acoustic encoder via its released code, or nothing at all.

The agent stage takes the conversation history as audio, the transcript, or both, and produces the next turn. It can optionally receive the perception record as text. The agents tested are a cascade (speech recognition followed by a text language model) and end-to-end speech language models of the kind evaluated in EchoMind.

The audit stage annotates each response, whether from the agent or the human, along two axes. The first is the emotion the response expresses. The second is its regulatory move: validating, exploring the cause, redirecting toward positivity, minimizing, or reframing. The move categories extend the support strategies of ESConv with explicit minimizing and redirecting categories taken from the Curry and Curry critique.

The central quantity is a suppression index. Let $s$ be the speaker state bin from the perception record, such as high-arousal anger. Let $m$ index the suppressive moves (minimize, redirect-to-positive). Let $p_A(m \mid s)$ and $p_H(m \mid s)$ be the rates of those moves for the agent and for IEMOCAP's human listeners in that bin. The index for bin $s$ is

$$\mathrm{SI}(s) = \sum_{m \in \text{suppressive}} \big(p_A(m \mid s) - p_H(m \mid s)\big),$$

and we report it alongside the corresponding shift in the response's expressed valence. A positive index means the agent smooths over negative states more than people did. A value near zero means it behaves like a human listener. The human reference has an obvious weakness: IEMOCAP listeners are actors, so their responses are not a clinical norm. We therefore restrict the reference to the improvised sessions and validate the index against fresh human ratings rather than treating the actors' choices as ground truth.

Conditioning on the speaker's state matters for a specific reason. Earlier work on emotion transitions between interlocutors, the EIT model of Zhang et al., shows that how one speaker's emotion follows another's is structured and informative on IEMOCAP. That work uses the structure to improve recognition. We use it as a behavioural norm against which agents are compared.

Regulatory lag enters as a design constraint rather than a modelling target. Rules are written as versioned predicates over the stored perception and audit records. An example rule: "if the speaker is in a high-arousal negative state, the response must not minimize before acknowledging the cause." Comparative work on governance documents (Luna et al.) found that only one of 15 governance processes is aligned across six jurisdictions and that rules decay over time. Khan et al. found that state AI bills are increasing rapidly but stall in specialist committees. Both findings suggest that the rules an empathetic agent must satisfy will keep shifting.

When a rule changes, we only re-run the predicate over stored records. The engineering claim is limited to this: the cost of a re-audit should scale with the number of records rather than with generation. Rule verdicts should also stay stable when an irrelevant stage, such as the speech recognizer, is swapped. Our rule versions are hand-written hypothetical amendments, not statutory text, and we say so.

Baselines

The naive reference point is an unconditioned positivity score. It measures how positive or calming the agent's responses are on average, ignoring the speaker's state and the human reference. If this crude number ranks agents the same way and agrees with human dismissiveness ratings as well as the suppression index does, then the conditioning and the human reference add nothing. We compute it on exactly the same responses.

The strongest competitor on the auditing question is a language-model judge. It receives the rule text and the conversation and is asked whether the response dismisses the speaker's feeling. We run two versions. One reads only the transcript. The other is an audio-capable judge that hears the speaker's turn. EchoMind found that GPT-4o-Audio did not average above 4/5 on its Speech Information Relevance score, which suggests that audio-capable judges may also underuse vocal cues. That remains a hypothesis to test, not an assumption. For whether such a judge detects suppression, there is no reported result on Lacuna.

For fairness, both judges see the same conversations and the same rule wording. They receive the same prompt-development budget we spend on the audit annotator: a fixed set of tuning dialogues, with no tuning on the test split. Beating them means significantly higher rank correlation with human dismissiveness ratings on the held-out set, plus a significantly larger advantage on the lexically-mild, acoustically-aroused subset.

For the perception stage, the strongest reported system on IEMOCAP is LFST+STEE at 87.5% accuracy and 86.8% F1 (Nfissi et al.). Older protocols report 63.89% unweighted accuracy (Lee and Tashev). The conversational system UniMEEC reports 74.83% weighted F1. These numbers come from different splits and label sets, so we re-run the perception options under leave-one-session-out on the improvised four-class subset before comparing anything. We do not need to beat these systems. We need to know how perception error propagates into the suppression index.

Two ablations isolate the key ingredient, acoustic grounding. In the first, the perception stage uses text only. In the second, oracle labels replace the perception stage. Comparing these shows how much of the index's validity depends on hearing the voice, and how much perception error costs.

Why not just…?

Why not just ask a strong model to judge empathy from the transcript? This is the real threat. If a transcript-only judge prompted with the rule matches the suppression index against human ratings, our main claim fails. We would report that, and the contribution would shrink to a cheap human-referenced metric. Busso and Rahman's activation result predicts the judge should fail specifically when arousal is vocal and the words are mild. The decisive experiment below targets that subset rather than the average.

Why not just run EchoMind? EchoMind already shows that speech models notice vocal cues and then fail to use them in their replies. But it uses semantically neutral single-turn scripts, mostly synthetic voices, and scores relevance, not direction. It cannot tell whether a response validated distress or talked the speaker out of it. It also cannot compare against what a human listener did in the same exchange. Our audit asks a different question on dyadic, multi-turn human speech, and EchoMind scores serve as a complementary column rather than a substitute.

Why not just bake the rules into the agent with reinforcement learning, as MULTIMOOD does? MULTIMOOD aligns a multimodal support agent to fixed psychological criteria judged by GPT-4o. That fixes behaviour against one rubric at training time. It does not measure suppression, and every rule change means retraining. This is precisely the lag problem. Our audit is complementary: it could serve as the reward, and it can check whether a MULTIMOOD-style agent still over-redirects relative to humans.

Proposed Experiments

The main study uses IEMOCAP's improvised sessions. We cut each dialogue after every speaker turn whose perception record falls in a negative bin (anger, frustration, sadness), at three arousal levels. The cascade and two or more open speech language models each generate listener turns, both with and without access to the perception record.

We collect human ratings of dismissiveness and of whether the response acknowledges the speaker's emotion. Raters judge about 600 agent responses and their paired human responses, hearing the audio. Each item gets three raters, and we report inter-rater agreement. The validation metric is Spearman correlation between each audit method and the mean rating.

The decisive check uses the subset where the text-based valence estimate is mild but acoustic arousal is high. The prediction is that the suppression index, using the acoustic perception stage, correlates with ratings significantly better than the transcript-only judge on this subset, and roughly equally elsewhere. If the advantage does not appear on this subset, the acoustic-grounding claim is wrong.

The attribution check feeds oracle labels to each agent. If the suppression index stays positive even with perfect perception, the happiness-by-default tendency comes from generation, not hearing. That would answer, for this behaviour, the question EchoMind left open about where the failure sits.

The lag check applies four hand-written rule versions to the stored records. It reports wall-clock time and cost for re-auditing against full regeneration plus re-judging. It also reports how often verdicts flip when only the speech recognizer is swapped. We expect near-zero flips, because the rule predicates should not depend on that stage.

A small transfer test applies the index unchanged to the CREMA-D-style acted single utterances. Without a human listener reference there, it reports only the unconditioned shift. This shows what is lost when the dyadic reference is removed.