Dialogue agents trained with reinforcement learning work better when they receive a reward at every turn from the user's emotional reactions, such as frustration in the voice or a smile on camera, rather than a single success signal at the end of the conversation. Current systems assume this affective signal is measured on every turn, for free, with no delay. In practice the richest signals come from audio and video models that are expensive to run, and many turns carry no new emotional information. We propose to treat the user's emotional state as a status that the system chooses to refresh. This is the framing of Age of Information, a networking field that studies how stale a receiver's view of a changing source is and when the source should send an update. A cheap text-sentiment model runs every turn. An expensive multimodal affect measurement runs only when the system's uncertainty about the user's emotion, accumulated since the last measurement, crosses a threshold. That uncertainty is propagated through a model of how the system's own actions push the user's emotion around.
The bet is that the turns where measurement matters most are the turns where the user's emotion changes. A user is calm for five turns, the system misunderstands a request, and the user becomes dissatisfied. Those change points are predictable in part from what the system just did. A measurement schedule that tracks accumulated, action-dependent uncertainty should keep almost all of the affective reward's training value at a fraction of the sensing cost. Uniform schedules and schedules driven only by per-turn text uncertainty should not. If that holds, multimodal affective rewards become affordable for deployed or large-scale RL, and the result also settles an open theoretical case in the freshness literature.
Core Claim
Fix a budget, for example 20% of turns receiving a multimodal affect measurement. Under that budget, a sampler that triggers on a generalized age of the affect estimate will recover most of the full-sampling benefit. Generalized age here means the conditional entropy of the user's emotional trajectory since the last measurement, given the observed text and the system's own actions. The sampler should recover both the reward signal and the speed with which the user's shifts into dissatisfaction are detected. It should do this measurably better than uniform sampling and than triggering on per-turn text-classifier uncertainty, and the advantage should be concentrated on emotional change-point turns.
The quantity we track adapts the generalized age of Information Freshness for Timely Detection of Status Changes. That paper defines freshness as accumulated conditional entropy since the last delivered update, $\Phi(n) = H(U_{\delta(n)+1:n} \mid U_{\delta(n)})$. It proves that this entropy-based age, unlike wall-clock age, stays affinely tied to the delay in detecting a change point in non-stationary processes. In our setting, $U_n$ is the user's discrete emotion at turn $n$ and $\delta(n)$ is the last turn with a multimodal measurement. We condition additionally on the cheap text signals $x$ and the system actions $a$:
$$\Phi(n) = H\big(U_{\delta(n)+1:n} \mid U_{\delta(n)},\, x_{\delta(n)+1:n},\, a_{\delta(n):n-1}\big).$$
The sampler measures at turn $n$ when $\Phi(n) > \lambda$. The multiplier $\lambda$ is tuned to meet the budget, which is the Lagrangian treatment of a constrained decision problem used for energy-limited sampling in Joint Status Sampling and Updating for Minimizing Age of Information in the Internet of Things.
Two things are new. First, the Feng and Yang equivalence was proven only for sampling schedules that ignore the process state, and adaptive, event-triggered schedules were left open. Our sampler is state- and action-dependent by construction, so the work tests empirically whether the age–detection-delay link survives. Second, no dialogue work selects which turns receive affective measurement. Every affective-reward system in the file measures every turn.
Research Program
The core object is a small emotion transition model $P(u_n \mid u_{n-1}, a_{n-1}, x_n)$ over emotion classes. It is trained on dialogues with emotion labels and run as a forward filter between measurements. It also yields the entropy $\Phi(n)$ by accumulating per-step conditional entropies along the filtered trajectory. The action term matters: system errors, repeated requests and failed bookings should raise the predicted probability of a negative transition, and with it $\Phi$. When a measurement arrives, the filter collapses onto it and $\Phi$ resets. This produces the sawtooth that Age of Information analysis studies, except that it rises in entropy per turn rather than in seconds.
On unmeasured turns, the affective reward is imputed from the filtered belief. Measured turns use the expensive signal. Both feed the same RL pipeline. Following the value-aware idea in Efficient Information Updates in Compute-First Networking via Reinforcement Learning with Joint AoI and VoI, which multiplies staleness by a consistency term so that updates are sent only when the receiver's view would actually be wrong, we also test a variant. That variant weights $\Phi$ by the expected change in reward magnitude a measurement would produce. Measurement then concentrates on turns where being wrong about emotion would change the training signal, not merely on turns where the label is uncertain.
We study three settings of increasing realism. The first is a controlled testbed on EmoWOZ. Its gold emotion labels play the role of the expensive sensor, and predictions from a text-only contextual classifier (the paper reports a weighted F1 of 79.7 for its ContextBERT model) play the role of the cheap signal. EmoWOZ suits this testbed because its human–machine portion records users turning dissatisfied after system failures, giving real change points. The second is DSTC1 with the reward setup of Sentiment Adaptive End-to-End Dialog Systems. There, TF-IDF and dialogic features are cheap every turn, and openSMILE acoustic features become the budgeted modality. The third is CANDOR with the pipeline of Global Reward to Local Rewards (GELI). GELI splits a post-conversation satisfaction score into turn rewards, using frame-level facial affect as local guidance. In our version the facial-affect guidance is available only on sampled turns.
Baselines
The naive reference points bracket the problem. Text-only never takes a multimodal measurement and uses the cheap signal throughout. It is the floor, and the file indicates it is a real floor: GELI beat text-only and global-only reward baselines on 6 of 8 conversational quality metrics on CANDOR, and the DSTC1 multimodal sentiment classifier (F1 0.686) outperformed its single-modality variants. Full sampling measures every turn and is the ceiling. On DSTC1 its downstream reference is the reported SRRIP convergent task success of 94.3% against 92.4% without sentiment reward, with a 5% lead after 5,000 iterations. Our target is to close most of the 92.4-to-94.3 gap at a fraction of the acoustic cost. All samplers share the cheap model, the transition model, the RL algorithm, the seeds and the hyperparameter search. Budgets are matched exactly by tuning each method's single threshold or rate.
Uniform sampling at the budget rate is the strongest classical competitor. Optimal Status Updating for an Energy Harvesting Sensor with a Noisy Channel proves that a best-effort uniform policy minimizes average age in its setting, so a schedule that ignores content could be competitive. Beating it means lower detection delay on dissatisfaction onsets and lower reward error at equal budget, across at least three budgets.
Text-uncertainty triggering measures whenever the cheap classifier's predictive entropy on the current turn exceeds a threshold. It is the most dangerous competitor, because it is a one-line value-of-information heuristic that needs no transition model.
An AVA-style trigger ports the age-times-consistency rule of the compute-first networking paper, using turns since last measurement times disagreement between cheap and last-measured affect. It is the strongest existing method for selective updating. That paper reports over 90% average reduction in update frequency (up to 98.34%) without loss of decision accuracy, in a single-source network simulation. There is no reported result on Lacuna for any selective-measurement method applied to affective dialogue rewards, so all baseline numbers in our tables will be our own runs.
Two ablations isolate our ingredients. No-action removes $a$ from the transition model, testing whether conditioning on the system's own behavior is what predicts change points. No-accumulation uses only the current turn's conditional entropy rather than the sum since the last measurement, which reduces our method to a filtered form of text-uncertainty triggering. If either ablation matches the full method, the corresponding claim fails.
Why not just…?
Why not just trigger on the text classifier's uncertainty? This is the real threat, and the proposal is built to be falsified by it. The argument against it is structural. A short utterance like "No, that's fine" can be classified confidently and wrongly; EmoWOZ notes exactly this kind of context-dependent ambiguity in task dialogue. Per-turn uncertainty also has no memory, so a run of mildly uncertain turns never triggers a measurement even though the belief has drifted far. Our method differs by accumulating uncertainty and by conditioning on system errors. The no-accumulation ablation and the head-to-head at matched budget settle it. If text-uncertainty triggering comes within seed noise of our sampler on onset-turn detection delay at every budget, we conclude that the freshness framing adds nothing in dialogue and report that.
Why not just sample uniformly, since uniform schedules are provably age-optimal? The optimality result in Optimal Status Updating for an Energy Harvesting Sensor with a Noisy Channel concerns average wall-clock age. Feng and Yang show that under change-point dynamics, wall-clock age decouples from detection delay, which is what matters for reward quality. Emotional shifts after system failures are change points, so uniform sampling should waste measurements on stable stretches and miss onsets. If the EmoWOZ onsets turn out to be rare or evenly spread, uniform will match us. We measure the onset distribution first so this outcome is interpretable.
Why not just run the affect model on every turn, or keep everything in text? Full sampling is our ceiling, not a rival. The question only exists where measurement has a cost: video and acoustic inference at scale, bandwidth to an edge server, or user consent to be watched. Joint Status Sampling and Updating for Minimizing Age of Information in the Internet of Things shows that measuring as often as possible is strictly suboptimal once cost is counted, cutting average age by up to 33% against that policy. Text-only is the floor the GELI and DSTC1 results already show to be lossy. The budget sweep reports where between floor and ceiling each method sits.
Why not just use delayed-reward RL? Methods such as Off-Policy Reinforcement Learning with Delayed Rewards and Belief Projection-Based Reinforcement Learning for Environments with Delayed Feedback compensate for a reward that arrives late. They do not decide whether a reward is measured at all. If measurements carry timestamps, late arrival is a bookkeeping problem. Our problem is the missing measurements in between, which such methods leave untouched. For this reason we deliberately do not build the proposal on reward discounting by age.
Proposed Experiments
On EmoWOZ we sweep budgets of 10%, 20%, 30% and 50% of user turns. The primary metrics are detection delay in turns from a gold dissatisfaction onset to the first measured or confidently imputed dissatisfied state, the fraction of onsets missed within three turns, and macro-F1 of the imputed per-turn affective reward against gold labels, reported separately on onset turns and stable turns. The decisive check is the onset-turn breakdown. Our claim predicts the gap over uniform and text-uncertainty triggering lives almost entirely there.
We also test the open theoretical question directly. Following Feng and Yang, we plot detection delay against average $\Phi$ for each adaptive sampler, and ask whether the affine relationship they proved for content-independent schedules persists for our action-dependent one.
On DSTC1 we retrain the sentiment-reward policy of Sentiment Adaptive End-to-End Dialog Systems with acoustic features available only on sampled turns. We report task success, iterations to reach the full-sampling curve, and average dialogue length against the 92.4% and 94.3% reference points. For online interaction beyond the fixed corpus, we use the langEmoUS emotional user simulator from Infusing Emotions into Task-oriented Dialogue Systems. Its success rates are 91.7% for the emotion-aware system versus 79.8% for the plain one in human trials. Policy gains here are secondary to the EmoWOZ signal-level results, because the simulator's emotional reactions are themselves model-generated.
On CANDOR we replace GELI's every-turn facial-affect guidance with budgeted guidance. We first evaluate at the reward-model level, measuring correlation of decomposed turn rewards with the full-sampling decomposition. We then run a reduced-scale PPO alignment of the text-only agent and score it with the paper's eight conversational quality metrics. The question is whether a 20–30% facial-affect budget preserves GELI's advantage over its text-only baseline.
The proposal fails cleanly in either of two cases. Text-uncertainty triggering may match our sampler on onset detection across budgets. Or the no-action ablation may match the full method, which would mean system behavior carries no predictive information about emotional change points. Either outcome is informative, and either would redirect the work toward simpler measurement heuristics.
References
- Information Freshness for Timely Detection of Status Changes
- Joint Status Sampling and Updating for Minimizing Age of Information in the Internet of Things
- Efficient Information Updates in Compute-First Networking via Reinforcement Learning with Joint AoI and VoI
- EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion in Task-Oriented Dialogue Systems
- Sentiment Adaptive End-to-End Dialog Systems
- Global Reward to Local Rewards: Multimodal-Guided Decomposition for Improving Dialogue Agents
- Optimal status updating for an energy harvesting sensor with a noisy channel
- Off-Policy Reinforcement Learning with Delayed Rewards
- Belief Projection-Based Reinforcement Learning for Environments with Delayed Feedback
- Infusing Emotions into Task-oriented Dialogue Systems: Understanding, Management, and Generation