Intent Tests for Emotional Side Effects in Support Dialogue

When a support chatbot asks a grieving user about the loss, the user's sadness often rises for a turn before it falls. Was that sadness something the system aimed for, or a cost it accepted on the way to relief? Today no one can say. Emotional-support dialogue systems predict a mixed emotional state for the user: a probability spread over categories such as sadness, anxiety and anger, rather than a single label. They then pick a support strategy, such as asking a question, reflecting feelings or giving suggestions. Nothing in these systems records which emotional changes a chosen strategy is for.

The philosophy-of-agency and agent-logic literature has a precise answer to "for": an outcome is intended if the agent would stop pursuing it once it is already achieved, and would re-plan if it fails to occur. An outcome the agent merely foresees passes neither test. We propose to run these two tests counterfactually on dialogue strategy planners, with the user's emotion distribution as the outcome variable. The result is an "intent profile" per strategy decision that separates targeted emotional shifts from collateral ones. We will validate it against planners whose intended targets we planted ourselves, and against human judgments.

Core Claim

A planner's intent toward an emotional outcome can be read off its behavior under two interventions on the dialogue history. The first is a satisfaction intervention: we edit the user's turn so that the emotion estimator reports the target shift as already achieved. The second is a frustration intervention: we edit the turn so that the shift visibly failed. This operationalizes two established ideas:

Neither paper handles outcomes that are distributions. That gap matters here, because an emotion distribution is never exactly "achieved." We borrow the approximate-correctness margin of Approximately Correct Label Distribution Learning, which treats a predicted distribution as correct when it lies within a divergence margin δ of the target. We use that margin to define when an emotional goal counts as satisfied.

For a dialogue history $h$, planner strategy distribution $\pi(\cdot \mid h)$, and emotion dimension $k$, we score intent as

$$I_k(h) = \mathrm{TV}\big(\pi(\cdot\mid h),\, \pi(\cdot\mid h^{\text{sat}}_k)\big) \cdot \Delta^{\text{track}}_k(h).$$

The symbols are defined as follows:

  • $\mathrm{TV}$ is total variation distance, a standard measure of how different two probability distributions are.
  • $h^{\text{sat}}_k$ is the history edited so that the estimated shift in dimension $k$ lies within δ of the planner's own forecast shift.
  • $\Delta^{\text{track}}_k(h)$ is the increase in probability of choosing the same strategy family again under the frustration edit $h^{\text{fail}}_k$.

A shift the planner's own forward model predicts, but for which $I_k$ is near zero, is classified as a foreseen side effect.

The Role of Logical Omniscience

Logical omniscience enters as a constraint on the analysis, not as a second mechanism. It is the idealization, built into standard modal logics of belief, that an agent believes every consequence of what it believes. Standard logics of belief, desire and intention close intentions under believed consequences. Chen et al. showed that this closure, a byproduct of the omniscience assumption, is exactly what forces agents to "intend" their side effects.

Two lines of work avoid this closure. Weak realism (Asymmetry Thesis and Side-Effect Problems) requires only that the worlds an agent believes possible overlap with the worlds where its goals hold, rather than being contained in them. Non-normal neighborhood semantics (A Modal Logic for Beliefs and Pro Attitudes) uses a semantics in which pro-attitudes are not closed under consequence. Both avoid closure by design, but both are static and hand-specified. Our test is behavioral and never closes under consequence.

We add a bounded notion of foresight in the spirit of Depth-bounded epistemic logic, which makes reasoning depth an explicit parameter. A consequence counts as "foreseen" only if the planner's explicit forward model produces it within a rollout of $d$ turns. This mirrors the explicit/implicit belief split of In Praise of Belief Bases, in which an agent explicitly believes only what is actually in its finite store of formulas. The bound gives a three-way classification: intended, foreseen but not intended, and not foreseen. The third category is one that omniscient causal accounts of blame cannot express. Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility and Moral Responsibility for AI Systems both assume the agent evaluates all counterfactual branches.

Research Program

Emotion estimator. We will build a turn-level estimator of the user's emotion distribution. It will follow EmoDynamiX's soft mixed-emotion representation with a learned temperature (EmoDynamiX), trained with the δ-LDL loss rather than Kullback-Leibler divergence (KL, the standard divergence between probability distributions). We make this choice because Li et al. show that KL-trained models can score worse than uniform guessing under label noise.

Choosing δ. The research file lists no text-emotion label-distribution benchmark. We will therefore set δ on held-out ESConv turns so that "achieved" matches annotator agreement, and report sensitivity across δ.

Counterfactual edits. We will generate satisfaction and frustration edits with a constrained rewriter. An edit is accepted only if two conditions hold. First, the estimator moves along the target dimension $k$ within δ. Second, other dimensions change by less than δ. This check keeps interventions targeted, which is the precondition the causal definitions require.

Planners analyzed. We will analyze three kinds of planner:

  • EmoDynamiX itself.
  • Prompted large language models.
  • A family of planted-intent planners. These are fine-tuned from EmoDynamiX with a reward on exactly one emotion dimension, in data where a second dimension is strongly correlated with it.

The planted planners are the dialogue analogue of the CoinRun goal-misgeneralization test in Ward et al. We know what each one intends, and a correlated side effect is present by construction.

Planner-side consequence. We will then test whether intent profiles support a practical change: adding a tracking term during training. When the targeted shift fails, the planner is penalized for abandoning the strategy family unless the estimator indicates that the goal became impossible, for example after repeated failure. This is the persistent-goal drop condition of Cohen and Levesque, which lets an agent abandon a goal only when it is believed satisfied or impossible.

Baselines

Naive reference points. The first is a uniform-random intent score. The second is attribution by marginal association: an emotion dimension is "intended" if its expected shift after a strategy is large. For strategy prediction, the corresponding floor is majority-class or uniform strategy choice.

Omniscient-closure baseline. Any shift the forward model predicts at unlimited depth is labeled intended. This is the normal-modal-logic reading that Chen et al. criticize. It shares our estimator and forward model, so it isolates the non-closure ingredient. Beating it means our method gives higher accuracy on planted-intent recovery specifically for the correlated side-effect dimension. There the closure baseline should, by construction, label the side effect as intended.

Discrete-label causal test. This is Ward et al.'s redundancy test applied to argmax emotion labels instead of distributions, with identical edits, rewriter and planners. It isolates the label-distribution ingredient. We expect it to fail when the intended and collateral emotions are the same argmax category at different intensities, and in mixed states. Beating it means higher agreement with planted ground truth on the subset of turns with non-dominant targets.

Redundancy-only ablation. This drops the tracking factor from $I_k$, to show whether both tests are needed.

Strongest strategy predictor. For strategy prediction, the strongest existing method is EmoDynamiX. It reports Macro-F1 of 27.70 on ESConv versus 26.28 for TransESC, and roughly 38% lower preference bias than TransESC (EmoDynamiX). We rerun it with the same splits, encoder and hyperparameter search budget as our tracking-augmented variant. Beating it means a Macro-F1 gain beyond seed variance and no increase in preference bias. The result that matters more is a tracking-failure rate reduced on frustration probes.

Human attribution reference. Mao and Gratch's attribution model reached an average Kappa of 0.732 with human judgments of intention and foreknowledge in scripted scenarios (Modeling Social Causality and Responsibility Judgment). This is not the same task, but it is the reference level for model-human agreement on attribution.

Why not just…?

Why not just ask the model what it intends? Self-report is the cheapest alternative, and for LLM planners we will run it. EmoDynamiX documents that LLMs over-choose strategies such as "Providing Suggestions" regardless of user state. That preference bias suggests stated rationale and actual behavior diverge. Ward et al. argue that an intent claim needs an intervention to be checkable, and their GPT-4 aspirin example is itself a behavioral probe, not a self-report. The planted-intent planners settle the question: if self-reported targets recover the planted goal as accurately as $I_k$, our test adds nothing for LLMs. EmoDynamiX-style planners cannot self-report at all, so the test would still be needed for them.

Why not just use correlation between strategies and subsequent emotion shifts? The planted planners are built to make this fail, because the side-effect dimension moves as reliably as the target. If correlational attribution matches $I_k$ on planted planners, the counterfactual machinery is unjustified and we will say so.

Why not just add next-turn emotion prediction as an auxiliary loss to EmoDynamiX? This is the real threat to the planner-side claim. An auxiliary target may deliver the same Macro-F1 gain as our tracking term. We therefore include it as a matched baseline. We make the diagnostic, not the F1 gain, the primary contribution. The tracking term is claimed useful only if it reduces abandonment on frustration probes beyond what the auxiliary loss achieves.

Why not formalize the planner in a non-normal modal logic directly? The logic of A Modal Logic for Beliefs and Pro Attitudes avoids side-effect closure but is static and single-agent, and it has no link to learned emotion distributions. Compiling constraints into tractable circuits, as Learning tractable probabilistic models for moral responsibility and blame does with probabilistic sentential decision diagrams, reintroduces classical deductive closure. A behavioral test sidesteps that tension.

Proposed Experiments

Planted-intent recovery (decisive check). Across planted planners with varying correlation strength between target and collateral dimensions, $I_k$ must recover the planted target, measured by area under the ROC curve. It must do so while the closure and correlational baselines mislabel the collateral dimension. It must also degrade gracefully as correlation approaches 1, the point where no behavioral test can separate the two dimensions.

Depth-bound check. For planners with $d$-turn lookahead, shifts appearing only beyond $d$ turns should be classified as not foreseen. Varying $d$ should move outcomes between the foreseen-side-effect and not-foreseen categories, and should leave the intended category unchanged.

Human validation on ESConv. We collect annotator judgments of intended versus accepted emotional consequences for sampled ESConv supporter turns. We then compare agreement for $I_k$, the discrete-label test and self-report, using Mao and Gratch's 0.732 Kappa as the reference level.

Transfer to AnnoMI. We repeat the strategy-prediction and frustration-probe comparisons on the AnnoMI counseling dialogues. The file reports no numerical AnnoMI result for EmoDynamiX, so we will report our own reruns.

What would falsify the proposal. The proposal fails if distribution-valued intent tests do no better than argmax labels or correlation on planted ground truth, or if human annotators do not distinguish intended from accepted emotional effects reliably enough to validate any method.