Telling Schema Ambiguity from Multifunctionality in Psychotherapy Coding

When two expert coders disagree about whether a therapist's sentence is a Clarification or an Interpretation, there are two very different explanations. The sentence might genuinely do both things, with one clause asking the client to say more and another clause offering a reading of their feelings. Or the two categories might be poorly separated in the coding manual, so that the same words reasonably fit either one. The first explanation calls for multi-label annotation. The second calls for fixing or merging the categories. Clinical dialogue NLP currently cannot tell these cases apart, so it defaults to treating all disagreement as noise. We propose to use per-label evidence extraction to answer the question one utterance at a time. We train a classifier that must justify each label with its own span of words, and we forbid those spans from sharing tokens. If both labels survive this constraint, the utterance is doing two things. If forcing the evidence apart destroys one of the labels, the two labels are competing for the same words, and the disagreement reflects the schema.

The motivating observation comes from Predicting Client Emotions and Therapist Interventions in Psychotherapy Dialogues. That work codes about 47,000 utterances from Hebrew psychotherapy sessions with one intervention label each. It reports that model confusion between Clarification and Interpretation mirrors human inter-rater disagreement, and it attributes this to schema ambiguity without testing the alternative. Work on multidimensional dialogue acts supports the alternative. The independence of dimensions in multidimensional dialogue act annotation argues that communicative functions operate along independent dimensions, and CAMAL shows that teachers routinely validate a claim and press for evidence in the same turn. Neither line of work gives a clinician, or a coding-manual author, a principled test for which explanation applies to a given disagreement. The interpretability contribution here is therefore not a nicer explanation for end users. It is a measurement instrument for the annotation scheme itself, in a domain where coding schemes drive clinical quality assessment and supervision.

Core Claim

Whether a pair of co-assigned labels reflects true multifunctionality can be estimated from how separable their evidence is. Concretely, we train a multi-label utterance classifier with one rationale extractor per label. We use the hard-deletion masking of MARE, in which tokens selected as evidence for one label are made invisible to every other label's attention head. For an utterance $x$ and candidate labels $a$ and $b$, let $r_a$ and $r_b$ be the extracted spans. Let $p(a \mid \cdot)$ be the classifier's probability for label $a$. We define an evidence-separability score:

$$ S(x; a, b) = \min\left( \frac{p(a \mid x_{r_a \setminus r_b})}{p(a \mid x)},\ \frac{p(b \mid x_{r_b \setminus r_a})}{p(b \mid x)} \right), $$

where $x_{r}$ denotes the utterance with only the tokens in span $r$ kept, in context. High $S$ means each label is supported by words the other does not need, which is what "doing two things" looks like. Low $S$ means at least one label collapses once it is denied the other label's evidence, which is what two overlapping category definitions look like.

The claim is falsifiable. $S$ should predict expert judgments of "does both" versus "unclear which" better than signals that ignore where the evidence sits. Those signals include raw annotator disagreement and the joint confidence of an ordinary multi-label classifier.

Research Program

The work has three parts.

Part 1: validate the separability diagnostic where ground truth exists. CAMAL provides 18,460 classroom utterances with expert multi-label annotation. There we know which utterances carry two moves and which carry one, including pairs of moves that are easily confused. This lets us test whether $S$ distinguishes genuine co-occurrence from confusable single-label cases before we touch clinical data.

Part 2: build a clinical test set. We re-annotate a stratified sample of the Hebrew therapist-intervention corpus. The sample is drawn from utterances where the original coders disagreed or where the single-label model confused intervention classes. Two clinically trained, Hebrew-speaking coders mark three things for each sampled utterance: every applicable intervention label, the supporting span for each label, and a judgment of whether the utterance "performs both functions," "fits either label equally with the same words," or "fits one label." Their spans also let us check extracted rationales against human evidence. Self-Verification reports 93% overlap with expert spans on clinical notes, which we take as a reference point for what is achievable.

Part 3: turn per-utterance diagnoses into schema revisions. We aggregate $S$ over each label pair. Pairs dominated by high-separability utterances are converted to multi-label coding. Pairs dominated by low-separability utterances are flagged for definition revision or merging. We then measure whether the revised scheme raises inter-coder agreement and model F1 on held-out sessions.

The encoder follows the adapter setup of the source paper (AlephBERT with bottleneck adapters, three-turn context) so that gains are not confounded with a bigger model. Context turns are available to every label head. Only the current utterance's tokens are subject to hard deletion. Two further design choices follow from constraining prior work. First, Rationale production to support clinical decision-making found that minimal extractive rationales from InfoCal lose accuracy relative to full transformers. For that reason we use rationales only to compute $S$, never as the classifier's sole input for the reported labels. Second, DILA showed that dense clinical features are polysemantic, with clinicians able to name their theme only 42% of the time versus 68% for sparse dictionary features. We therefore include a sparse-feature variant of the separability test as a secondary analysis, not as the core method.

A known failure mode needs stating up front. Some functions are carried by the same tokens through tone. Investigating the role of 'yeah' in stance-dense conversation shows a single word layering agreement and reluctance through duration and pitch. A token-disjointness test will misread such cases as ambiguity. We therefore report results separately for utterances under five tokens and treat that stratum as outside the method's scope.

Baselines

Naive reference point. Treat every coder disagreement as schema ambiguity, which is the implicit current practice. As an alternative naive reference, label every disagreement as multifunctional. Both predictors yield chance-level AUROC on the three-way expert judgment by construction. They matter because they fix the operating cost of doing nothing: either every disputed pair gets merged, or every one gets double-coded.

Label-probability baseline. Train a plain multi-label classifier on the same encoder, data, and tuning budget, and score each utterance by the product of its two label probabilities. This is the strongest cheap competitor, because it captures "the model thinks both apply" without asking where the evidence is.

Strongest existing classifiers. These establish that the diagnostic does not come at the cost of classification. For single-label intervention prediction, the source paper reports micro F1 of 0.64 with adapters (Predicting Client Emotions and Therapist Interventions in Psychotherapy Dialogues). We require our multi-label model's top-1 prediction to stay within one point of that figure on the original labels. On CAMAL, the strongest reported multi-label system is the CNN+LSTM+Speaker-ID-graph model at 65.1% F1, against 78.2% for humans (CAMAL). We rerun that model as a classification reference. We do not expect separability scoring to change its F1.

LLM evidence baseline. Adapt the four-stage Self-Verification pipeline (extract, re-check for omissions, ground each label in a verbatim span, prune contradicted items). It is run with a frontier multilingual model on the same sampled utterances, and its per-label spans are fed into the same $S$ computation using overlap of spans instead of masked probabilities. This is the most serious rival, because it also yields per-label evidence.

Ablations that isolate the key ingredient. The first replaces hard deletion with ordinary soft attention rationales, which allow evidence to bleed across labels. The second computes spans by SHAP attribution on the plain classifier.

For the diagnostic itself, no reported result was found on Lacuna for any method, on any corpus. Beating the baselines therefore means a pre-registered margin: at least 0.05 AUROC over the label-probability baseline and the soft-attention ablation on both CAMAL and the clinical test set.

Why not just…?

Why not just ask coders to re-annotate everything in multi-label form? Expert re-annotation is the ground truth, and we collect it for validation. It does not scale to tens of thousands of utterances, however. Multi-label coding of a whole corpus also does not by itself say which label pairs are ill-defined, because coders who find two categories interchangeable will often tick both. The coders' third answer option ("either label, same words") exists precisely to separate these cases. The separability score is useful only if it reproduces that judgment cheaply, and Part 2 tests exactly that.

Why not just train a multi-label classifier and read off which pairs co-occur? This is a real threat. The label-probability baseline tests it directly. The argument against it is that co-occurring high confidence is what both explanations predict: an ambiguous pair also receives two high scores, from the same words. If evidence location adds nothing beyond joint confidence, the AUROC comparison will show it, and we would report the diagnostic as unnecessary.

Why not just prompt a large model once for labels plus justifications? Self-Verification reports that single-pass GPT-4 extraction reaches F1 0.487 on MIMIC-IV ICD-10 coding versus 0.533 with verification, and 0.419 versus 0.530 on trial arms. AICare documents that ungrounded narrative LLM rationales produce plausible unsupported claims. More fundamentally, a generated justification can cite the same clause for both labels without any test of whether each label needs it. That is why we include the verified LLM pipeline as a baseline instead of dismissing it. If it matches the hard-deletion model, the cheaper-to-deploy option wins.

Why not use attention weights or SHAP for the spans? Rationale production to support clinical decision-making finds that attention spreads importance across the sequence rather than yielding a minimal explanation. DILA notes that KernelSHAP can take over an hour per note in high-dimensional multi-label coding. Utterances are far shorter than discharge notes, so cost is less decisive here. The two ablations measure whether leak-free evidence matters.

Proposed Experiments

The first decisive check is on CAMAL. Utterances annotated with two moves should score higher $S$ than single-label utterances drawn from the same confusable move pairs. We report AUROC against the label-probability baseline and both ablations.

The second check is the clinical test set. We report AUROC of $S$ for the expert "performs both" versus "same words, either label" judgment, and token-level F1 of extracted spans against coder spans. MARE's 88.8% token F1 on BeerAdvocate indicates the ceiling in easier monologue text.

The third check is consequential. After revising the scheme by the aggregated diagnosis, we recode held-out sessions and compare inter-coder agreement, and multi-label F1, against the original single-label scheme. As a secondary validity probe, the source paper correlates predicted client emotion with mood self-reports (r = 0.27 for positive, 0.21 for negative). We test whether session-level rates of diagnosed multifunctional therapist utterances relate to those same client reports. We treat any such relation as exploratory.

The idea fails if $S$ does not beat joint label confidence on both datasets. It is confirmed if it does, and if schema revisions guided by it raise coder agreement on held-out sessions.