When a customer says "I placed my order last week," a sentiment model usually labels it neutral. Sometimes that is right. Often, in the middle of a conversation about a late delivery, it is a complaint. Current systems have no principled way to tell these apart. Hand-engineered dialogue systems handle the problem with manually written escalation rules ("if the user restates the order date after a delay message, flag frustration"). Automatic methods that extract a conversation's flowchart from logs ignore emotion entirely. This proposal uses a simple observable to sort "neutral" utterances into kinds: when we sample a language model's judgment many times, does it agree with itself, and does adding a compact summary of where the utterance sits in the conversation's workflow turn disagreement into agreement?
The idea in two sentences: sample many independent reasoning chains for each user turn, once with no context and once given its position in an automatically induced dialogue flowchart, and measure how strongly the sampled labels agree each time. Utterances that are unanimously neutral without context are factual. Utterances that are scattered without context but converge once the flow position is given are context-dependent implicit sentiment. Utterances that stay scattered either way are intrinsically ambiguous. That three-way split lets us attach affect to the edges of an induced flowchart, which replaces a piece of manual dialogue engineering that no automatic structure-induction method currently addresses.
Core Claim
The neutral class is a catch-all. It mixes factual statements with balanced or unclear evaluations (IIIT-H at SemEval 2015). Classifiers trained on clean data fall back to the neutral or objective label on exactly the disputed items: Macro-F1 drops from about 0.80 on unanimous tweets to 0.53 on majority-agreement tweets, and to 0.163 on the COMPLICATED class (Sentiment Analysis: It's Complicated!). These two papers supply the motivation.
Self-consistency decoding samples several reasoning chains at nonzero temperature and takes a majority vote (Self-Consistency Improves Chain of Thought Reasoning). The level of agreement among samples has been shown to be a better confidence signal than token probabilities (Self-Consistency Boosts Calibration for Math Reasoning). That evidence comes only from math and factoid QA. That paper also reports that counting distinct answers fails on few-option tasks, while the share of samples agreeing with the winner ("cluster size") holds up. We reuse cluster size as our measurement.
Our claim is that, for dialogue, the informative quantity is the change in agreement when workflow context is added, not agreement alone:
$$\Delta(u) = F_{CS}(u \mid s) - F_{CS}(u),$$
Here $F_{CS}(u)$ is the fraction of $N$ sampled chains agreeing with the plurality label for user turn $u$ in isolation, and $F_{CS}(u \mid s)$ is the same fraction when the prompt also includes $s$, the turn's induced flow state. The flow state is the sequence of workflow nodes visited so far, plus flags for loops and regressions. We predict three outcomes. Factual neutrals have high $F_{CS}(u)$ and $\Delta \approx 0$. Implicit complaints have low $F_{CS}(u)$ and large positive $\Delta$. Intrinsically ambiguous turns have low agreement in both conditions.
The nearest prior work differs on two axes. Triple Reverse Chain Reasoning (Application of Multiple Chain-of-Thought in Contrastive Reasoning for Implicit Sentiment Analysis) forces the model to argue positive, negative, and neutral hypotheses and then arbitrates between them. It works on isolated SemEval-2014 sentences, never reads dispersion as a signal, and never uses dialogue context. Topic-aware multi-task learning (Sentiment Classification in Customer Service Dialogue with Topic-Aware Multi-Task Learning) shows that context resolves apparent neutrality. It uses latent topics in a discriminative classifier, with no explicit workflow and no notion of which turns context actually resolves.
Research Program
The first component induces the flowchart without manual labels. We embed utterances with the action-oriented sentence encoder of Dialog2Flow, which groups utterances by conversational function rather than topic. On SpokenWOZ it recovered graphs within 8.5% of human complexity, where Sentence-BERT missed about half the steps. We cluster the embeddings into action nodes and build the transition graph from the conversation trajectories. Dialog2Flow also constrains our design: its embedding is built to discard surface phrasing, which is where affect often lives. So we use it only to locate the turn in the workflow. Affect judgments come from the language model reading the raw utterance.
The second component is the sampling protocol. For each user turn we draw $N$ chains at temperature 0.7 under two prompts, "utterance only" and "utterance plus flow state," and record the sentiment label from each chain. We also run a third prompt, "utterance plus raw dialogue history," as a control. We will sweep $N$ from 4 to 32. Gains in the calibration study plateaued between 8 and 16 samples, so we expect $N \approx 8$ to suffice.
The third component builds the affect-annotated graph. We label each turn factual, context-resolved, or ambiguous using $F_{CS}(u)$ and $\Delta$ thresholds set on a development split. We then attach to each graph edge the rate at which context-resolved negative turns follow it. One example would be a user restating an inform after a "no availability" system act. The result is an automatically produced counterpart of hand-written escalation rules. It also addresses the open problem, noted for Dialog Flow Induction for Constrainable LLM-Based Chatbots and TOD-Flow, that induced flows model only functional acts.
The fourth component is ground-truth collection, since no file dataset labels context-dependence. We will annotate about 2,000 user turns from MultiWOZ and SpokenWOZ, with five annotators per turn. Each turn is labeled twice, once in isolation and once with the full dialogue, by different annotators. A turn is "context-resolved" if annotator agreement rises substantially when context is shown. This follows the disagreement-tier design of the MTSA tweet corpus (Sentiment Analysis: It's Complicated!), extended to a with/without-context contrast.
Baselines
The naive reference is a single greedy decoding of an instruction-tuned model that outputs one sentiment label per turn with full history in the prompt. This is what a practitioner would deploy first. It is run on the same model, prompt template, and turns as our method. Beating it means our three-way split identifies context-resolved and ambiguous turns with clearly higher AUROC against annotator ground truth, since greedy decoding yields no graded signal at all.
The strongest existing context-aware classifier is topic-aware multi-task learning, which reports 75.9% Macro-F1 on its 68,000-utterance Chinese e-commerce dataset, against 72.3% for vanilla BERT and 73.9% for DialogueRNN (TML). It is the right competitor for label accuracy because it is built for exactly the "neutral in isolation, negative in context" phenomenon. If that dataset is obtainable, we compare 5-point Macro-F1 on its split under its protocol. Beating it would mean exceeding 75.9% with plurality-vote labels from the flow-state condition. Our main claim does not require this, but it tests whether the measurement comes at a cost in accuracy. If the data cannot be obtained, we reimplement TML on our annotated MultiWOZ/SpokenWOZ turns with the same training data our threshold tuning uses.
The strongest multi-path sentiment method is Triple Reverse Chain Reasoning (TRCR). It reports about 6% F1 over THOR on SemEval-2014 implicit sentiment with Flan-T5-xxl; the file gives no absolute number. We run it on the same backbone, with matched total generated tokens: three forced chains plus arbitration, against $N$ sampled chains. We give it the flow-state context as well, for fairness. Beating it means better separation of the three neutral subtypes at equal token budget.
For the outcome-level comparison, the hand-engineered reference is the 24-act customer-service taxonomy with a sequential SVM-HMM (How May I Help You?), which reports 0.83 accuracy for predicting customer frustration. On that Twitter dataset, matching 0.83 with an induced, unlabeled flowchart plus our affect annotations would show that the manual taxonomy can be replaced. Exceeding it would be a bonus.
Two ablations isolate the key ingredients. The first swaps the induced flow state for raw history, which tests whether workflow abstraction adds anything beyond context. The second swaps sampled agreement for single-pass label-token entropy, which tests whether sampling is necessary.
Why not just…?
Why not just give the model the full dialogue history and decode once? This may get most of the label accuracy. It gives no way to know which neutral labels are trustworthy, and it produces no structure that can be attached to a workflow. The raw-history ablation measures this directly. If raw history matches the flow state on every metric, we will report that the workflow is useful only as an index for building the affect-annotated graph, not as a disambiguation signal.
Why not just read the softmax entropy over the three label tokens instead of sampling? This is the most serious threat. The calibration result showing logits losing badly (ECE 0.394 against 0.092 for sampled agreement on GSM8K with Mistral-7B, Self-Consistency Boosts Calibration for Math Reasoning) concerns long numeric reasoning, not one-token classification, where label probabilities may be far better behaved. We therefore compute the same with/without-context difference using single-pass entropy and compare AUROC against annotator ground truth. If entropy matches sampling, we adopt the cheap version and the contribution becomes the context-difference diagnostic and the affect-annotated graph, not the decoding method.
Why not just fine-tune a four-class classifier with a COMPLICATED label? That route already exists and reaches 0.163 F1 on the hard class (Sentiment Analysis: It's Complicated!). It also cannot separate "ambiguous in isolation but resolved by context" from "ambiguous regardless," because it sees one input condition. We include a fine-tuned classifier with dialogue context, trained on our annotations, as an additional comparison.
Why not just use Sentence-BERT clusters for the flowchart? Dialog2Flow reports Sentence-BERT at 33.30 F1 against 70.94 in 5-shot action classification, and missing about half of SpokenWOZ workflow steps (Dialog2Flow). We still run the Sentence-BERT flow state as a cheap ablation, to check whether coarse context is enough for affect.
Proposed Experiments and Decisive Checks
The first check validates the measurement on isolated text. On the MTSA tweets, isolated $F_{CS}$ should decrease monotonically across the unanimous, consensus, majority, and disputed tiers, and should be lowest on COMPLICATED. If sampled agreement does not track human disagreement here, the premise fails and we stop.
The second and central check runs on our annotated dialogue turns. The context difference $\Delta$ should separate context-resolved turns from both factual and intrinsically ambiguous turns, measured by AUROC against the annotator-derived labels. It should do so better than single-pass entropy differences and than TRCR at equal token budget. We report results across $N$ to locate the cost plateau.
The third check is downstream. On the Twitter customer-service data, we induce the flowchart without labels and attach affect edges. We then predict conversation-level frustration and compare against the 0.83 accuracy of the hand-engineered pipeline. On MultiWOZ we report whether affect edges concentrate after specific system acts, such as no-availability or error turns. A human audit will check whether the top-ranked edges correspond to plausible escalation rules.
The fourth check covers label accuracy. We report Macro-F1 against TML where its data is available, to confirm that the diagnostic does not cost accuracy.
Each outcome has a fallback. If the second check succeeds but entropy matches sampling, the paper stands with the cheaper estimator. If the flow state adds nothing over raw history, the contribution narrows to the affect-annotated graph.
References
- IIIT-H at SemEval 2015: Twitter Sentiment Analysis - The Good, the Bad and the Neutral!
- Sentiment Analysis: It's Complicated!
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Self-Consistency Boosts Calibration for Math Reasoning
- Application of Multiple Chain-of-Thought in Contrastive Reasoning for Implicit Sentiment Analysis
- Sentiment Classification in Customer Service Dialogue with Topic-Aware Multi-Task Learning
- Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction
- Dialog Flow Induction for Constrainable LLM-Based Chatbots
- TOD-Flow: Modeling the Structure of Task-Oriented Dialogues
- "How May I Help You?": Modeling Twitter Customer ServiceConversations Using Fine-Grained Dialogue Acts