Handing Off With a Reason: Description-Grounded Handoff Prediction for Chatbots

When a customer-service chatbot hands a conversation to a human agent, current systems make a yes/no call. They decide that the dialogue is "transferable" and say nothing about why. The human then has to reread the whole conversation to find out whether the customer is angry, whether the request falls outside what the bot can do, or whether the bot is stuck repeating itself. This proposal replaces that empty yes/no output with a small set of handoff reasons written in plain language, such as "the customer is repeating a request the bot has already failed to answer" or "the customer asks for a refund decision the bot is not authorized to make." At every turn, the model scores how well the conversation matches each written reason. It hands off when any reason is strongly matched, and it passes the matched reason to the human as context.

The part that makes this new is the training signal. The model learns only from the binary handoff labels that existing datasets already provide. The reasons appear only as text descriptions. They act as an interpretable bottleneck, meaning every handoff decision must pass through one of the named reasons. Because the reasons are text, a new failure type can be added by writing one sentence, with no retraining. If this works, a single design gives three things that are currently handled separately or not at all: the handoff trigger, the explanation, and the context handed to the human.

Core Claim

Handoff prediction on the standard Taobao machine-human chatting handoff benchmarks can be recast as matching each turn against written descriptions of reasons to hand off, plus reasons not to. This recasting should keep or improve handoff F1 relative to the best discrete-head models while doing three further things. It should produce a reason that trained annotators agree with. It should cut unnecessary handoffs by modeling confusable "looks bad but is fine" situations explicitly. And it should extend to newly described reasons zero-shot.

Concretely, let $h_t$ be the encoder representation of the dialogue up to turn $t$. Let $d_1,\dots,d_K$ be written reason descriptions with text embeddings $e_k$, and let $s(h_t, e_k)$ be a learned compatibility score. The handoff probability is a noisy-OR over reasons:

$$p(\text{transfer}\mid t) = 1 - \prod_{k=1}^{K}\bigl(1 - \sigma(s(h_t, e_k))\bigr),$$

where $\sigma$ is the logistic function. The noisy-OR fires if any single reason fires. Training uses binary cross-entropy on $p$ alone, so reason-level supervision is never required. We also add "counter-reasons" $\bar d_j$, such as "the customer is upset but the issue has just been resolved." These are scored the same way and multiply the transfer probability down. This is the mechanism aimed at the invalid handoffs that CEM attributes to surface sentiment cues.

Where This Sits

The handoff side supplies the task, the data, and the target to beat. Time to Transfer defines the task and its DAMI model, which uses difficulty features and loop detection over binary normal/transferable labels. RSSN models handoff jointly with satisfaction using speaker-role masking. CEM adds causal sentiment masking and a counterfactual cost simulator to penalize invalid transfers. All three keep a semantically empty output. We keep their input-side ideas, such as role filtering and loop matching, where helpful, and change only the output space.

When and Who? is the closest work that attaches a category to a handoff. It classifies dialogues into 8 fixed types and triggers on a hand-set threshold (predicted satisfaction below 4.0 and at least 8 turns). Its categories are index labels trained with cross-entropy, and they do not drive the trigger. In our model, the reasons are the trigger.

Kraus et al. motivate the context-transfer half. They argue that handing over control without handing over context creates delay for the human, but they offer no mechanism. The traceability arguments in the variable autonomy editorial and Basilico et al. motivate evaluating the reason itself, not just the timing of the handoff.

The label-embedding side supplies the method and its known constraints. Sainz et al. show that writing relation labels as short templates and scoring them with an entailment model gives strong zero-shot relation extraction. They also warn that results depend heavily on template wording, so we test wording robustness directly. DELE shows that description-aware contrastive training beats index heads and that input-conditioned attention can filter noisy descriptions. We reuse its sentence-to-label attention as the compatibility function $s$.

StructCBM is the source of the prototypical-versus-discriminative concept split and of an LLM-driven loop that refines concepts. Our counter-reasons are its discriminative concepts, applied to confusable handoff situations. Unlike StructCBM, which works on static single-turn classification with 10 examples per class, we train end to end on sequential dialogue. That setting introduces class imbalance and timing, which StructCBM never faced.

Research Program

The work has four parts.

The first part builds the reason library. From a few hundred transferable utterances in the Clothing training split, an LLM proposes candidate reasons and counter-reasons. Two annotators consolidate these into roughly 8–15 reasons and 3–6 counter-reasons, written in Chinese to match the data. This one-time step mirrors StructCBM's offline concept construction. The refinement loop then rewrites a description when validation errors cluster on it.

The second part is the model: a Chinese pretrained encoder with RSSN-style role-aware dialogue context, DELE-style attention between each turn and each description, and the noisy-OR head. The head is trained on binary labels only, with a small entropy penalty that pushes each positive turn toward one dominant reason.

The third part builds an evaluation-only reason annotation. About 500 transferable turns from each of the Clothing and Makeup test sets are tagged by two annotators with the best-matching reason, or "none of these." We report inter-annotator agreement. These tags are never used for training.

The fourth part is a small operator study. Staff with customer-service experience receive handed-off dialogues with the model's reason, with an LLM-written summary, or with nothing. They are asked to state the customer's problem and the next action. We measure time and correctness. This operationalizes the context-transfer claim of Kraus et al..

Baselines

The naive reference points are a fine-tuned BERT classifier with a binary head, and the same classifier with a confidence threshold tuned on validation. Time to Transfer reports roughly 56–57% F1 for BERT on Clothing and Makeup. This number shows how much structure matters beyond a pretrained encoder.

The strongest published discrete-head models are DAMI, RSSN, and CEM-enhanced RSSN. The file reports DAMI at 67.19% F1 on Clothing and 69.26% on Makeup, with GT-III tolerance accuracy up to 85.51% (Time to Transfer). It reports RSSN at 69.2% handoff F1 on the 10,000-dialogue Clothes set (RSSN). It reports CEM-RSSN at 64.8% F1 on Makeup2 versus 63.5% for plain RSSN, and it reports CEM lowering invalid cost on Clothing from 59.0% to 53.4% (CEM).

These papers use different dataset versions: 3,500 versus 10,000 Clothing dialogues, and Makeup versus Makeup2. We therefore re-run all three on identical splits, with the same encoder where the architecture allows, the same hyperparameter search budget, and five seeds. We quote the published figures only as sanity checks. Beating the strongest competitor means matching CEM-RSSN's F1 within seed noise while lowering invalid cost below it, or exceeding its F1 at equal invalid cost.

The key ablation isolates the written descriptions. We keep the same reason set but give each reason a learned vector initialized randomly rather than from its text, which turns the head into a multi-head index classifier. If this ablation matches our model on F1, reason agreement, and zero-shot extension, the language grounding is doing nothing. A second ablation removes the counter-reasons, to test whether they, rather than the positive reasons, drive any reduction in invalid cost.

For explanations and context transfer, the competitor is RSSN plus an LLM that writes a post-hoc explanation of each handoff.

Why not just…?

Why not just keep the best binary model and ask an LLM to explain each handoff afterward? This is the most serious threat, because it could match on timing and produce fluent rationales. Its weakness is that the explanation is not what caused the decision, so it can confidently explain false alarms. We test this directly. On invalid handoffs, the post-hoc explainer should still produce a plausible reason, whereas our model's top reason score should be low or dominated by a counter-reason. The operator study compares the two on correctness, not fluency. If post-hoc explanations are just as accurate on the annotated subset and just as useful to operators, we will report that the bottleneck buys nothing for context transfer, and the remaining contribution rests on invalid cost and zero-shot extension.

Why not just prompt an LLM at every turn to decide whether to transfer and why? Unlike surgery, chat does not impose millisecond deadlines, so latency is a weaker objection here. The real concerns are cost per turn at Taobao volume and the fact that the classes are heavily imbalanced, so a zero-shot model's operating point is hard to tune. We include a zero-shot and a few-shot LLM baseline on the same test sets, with a validation-tuned probability threshold. We report F1, invalid cost, and cost per thousand dialogues. If the LLM wins on F1, the case for our model reduces to cost and to a distillation target, and we will say so.

Why not just annotate reasons and train an ordinary multi-class head? That approach needs reason labels on every training turn, and it cannot absorb a new reason without relabeling and retraining. Our model needs neither. The random-vector ablation above is the same design with the text removed, so if it performs equally well, this objection holds.

Proposed Experiments

The main comparison uses F1, GT-I/II/III tolerance, and invalid cost on Clothing and Makeup with shared splits, against the baselines above.

Reason faithfulness is the agreement between the model's top reason and the annotated reason on the held-out tagged turns. We compare it with the agreement between the two annotators, and with the post-hoc LLM explainer.

Zero-shot extension works as follows. For each reason in turn, we delete its description, retrain, and then reinsert the description at test time. We measure how much recall on that reason's annotated turns recovers without any gradient update. The random-vector ablation cannot recover by construction, so the measure of interest is how close our model gets to the model trained with all reasons.

Wording robustness addresses the template sensitivity reported by Sainz et al.. We have three independent writers rephrase every description and report the spread in F1. Cross-domain transfer trains on Clothing and tests on Makeup, keeping or rewriting the descriptions.

The decisive checks are three. First, our model must stay within seed noise of CEM-RSSN on F1 on shared splits. Second, its reasons must agree with annotators clearly above the post-hoc explainer. Third, reinserting a held-out description must recover a substantial share of that reason's recall where the index ablation recovers none. Failing the first check means the bottleneck costs too much accuracy. Failing the second means the reasons are decorative. Failing the third means language grounding adds nothing over annotated categories.