Do Ensembles Amplify Speaker Bias in Few-Label Dialogue Act Classifiers?

A dialogue act classifier labels each utterance by what it does: a question, an agreement, a backchannel such as "uh-huh," an apology. These labels feed conversation analysis, annotation pipelines and dialogue agents, and labeled data for them is scarce. Two standard tools for small-label settings are self-supervised pretraining and ensembling, which averages several models so their separate mistakes cancel. Neither has been checked for a basic fairness failure. A classifier trained on skewed data can learn that certain speakers "tend to" backchannel or agree, then predict those acts for those speakers more often than the data warrants. This is bias amplification. Theory suggests that this amplification and the benefit of ensembling peak in the same place: where the number of model parameters roughly matches the number of labeled examples. That is exactly the regime a few-label classifier lives in.

The proposal makes one measurable claim. Whether an ensemble narrows or widens the error gap between speaker groups is fully determined by how much its members disagree within each group. Disagreement can be computed on unlabeled utterances, so a practitioner could predict before spending labels whether ensembling will reduce or worsen a group disparity. We test this on the Switchboard Dialogue Act corpus, using its recorded speaker metadata. We also separate the part of the amplification that simple threshold calibration removes from the part that only variance reduction can touch.

Core Claim

Three papers set up the question without answering it. When are ensembles really effective? shows that ensemble improvement tracks the Disagreement-Error Ratio, with a linear fit of R ≥ 0.96 on GLUE, IMDB and CIFAR. The Disagreement-Error Ratio is the expected disagreement between members divided by their average error. That paper also finds the gains vanish once models interpolate the training set, but it measures only overall accuracy. An Effective Theory of Bias Amplification proves that the gap in difficulty between groups peaks at the interpolation threshold, where parameters equal samples. The peak appears even with perfectly balanced 50/50 groups whenever their label noise differs. That analysis covers single linear and random-feature models and leaves deep NLP models and ensembles open. Directional Bias Amplification shows that much measured amplification comes from uncalibrated decision thresholds: recalibration drops it from 0.040 to 0.001 on their benchmarks. It also notes that seeds with equal accuracy have different fairness profiles, but it never ensembles those seeds.

Our contribution joins these at the group level. Let $e_g$ be the average member error on speaker group $g$ and $\mathrm{EIR}_g$ the ensemble's relative error reduction on that group. Let $\Delta = e_{1}/e_{2}$ be the member-level disparity ratio between groups 1 and 2. The ensemble's disparity ratio is then exactly

$$\Delta_{\text{ens}} = \Delta \cdot \frac{1-\mathrm{EIR}_{1}}{1-\mathrm{EIR}_{2}}.$$

This is an identity, so the content of the claim lies in its second half. We conjecture that each group's improvement rate is predicted by its own disagreement ratio $\mathrm{DER}_g$, following the linear relation established for overall accuracy. If that holds, the ensemble shrinks a disparity exactly when members disagree relatively more on the disadvantaged group. The numerator of $\mathrm{DER}_g$ needs no labels. We further predict that this group-level variance cancellation is largest near the interpolation threshold and is orthogonal to threshold calibration. Calibration fixes base-rate miscalibration, while ensembling fixes noise-driven variance, so the two should stack rather than substitute.

Research Program

The testbed is the Switchboard Dialogue Act corpus, made of spontaneous two-party telephone calls with a collapsed act taxonomy. Switchboard's speaker tables record sex, birth year, dialect region and education. We use two grouping axes. Speaker sex gives nearly balanced groups, which tests the theory's prediction that amplification occurs even at 50/50. Dialect region gives imbalanced groups, the conventional minority setting. The act-by-attribute statistics, for example how often each group produces backchannels or agreements, define the base rates against which we measure directional amplification. One amplification direction, which we call attribute-to-task, asks whether speaker group shifts predicted acts beyond the base rate. A secondary replication uses the emotional dialogue act corpora built on IEMOCAP and MELD in EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators, with speaker identity as the grouping.

The label-efficiency axis is controlled precisely. We freeze conversational text encoders and train linear probes on top. For a probe on a $d$-dimensional embedding with $n$ labeled utterances, the parameter-to-sample ratio $\psi = d/n$ is explicit. Sweeping $n$ from about $0.1d$ to $50d$ traces the regime in which the effective theory places its amplification peak. A second track fully fine-tunes the encoders at the same label budgets, where models interpolate easily and the ensembles paper predicts disagreement collapses.

The ensembles span three diversity sources described in the ensembling literature. Seed and bootstrap ensembles of probes vary only the data view. Heterogeneous-encoder ensembles pair RoBERTa- and ELECTRA-style models, as combined in Technical Report on Shared Task in DialDoc21. Utterance-plus-context ensembles mirror the DANA design of EDA, which mixes utterance-level and three-turn context models.

For every configuration we record four quantities: group-conditional error, group-conditional disagreement, $\mathrm{EIR}_g$, and directional amplification before and after per-group threshold calibration. To test the noise mechanism directly, we inject controlled label noise into one balanced group's training labels. We then check whether the disparity peak appears at $\psi \approx 1$ and whether ensembles flatten it.

Baselines

The naive reference point is a single model at the same label budget. Its disparity and amplification numbers are the quantities every intervention must improve on. Below it sits a majority-class predictor that defines chance-level error per group.

The strongest existing dialogue act ensemble is DANA from EDA. It arbitrates five recurrent annotators by unanimity, then context-model precedence, then softmax rank, and reports 77.8% accuracy on the Switchboard Dialogue Act test set with full labels. We reimplement its arbitration rule over our members, which gives a like-for-like comparison against averaging. We report its accuracy at full labels against that 77.8% figure. For the few-label regime, no reported result was found on Lacuna for Switchboard dialogue act accuracy at 1–10% labels, nor for any group-disparity metric on this corpus, so we establish those numbers ourselves.

The strongest fairness competitor is per-group threshold calibration from Directional Bias Amplification, which cut amplification from 0.040 to 0.001 in that paper. We apply it to the single model using a held-out calibration split that is counted against the label budget.

A compute-matched single model gets the same total parameters or training FLOPs as the K-member ensemble, through a wider probe, a larger encoder, or K-times-longer training. A prompted large-model classifier serves as a label-free reference. It is motivated by the GPT-4 act classifier in Grounding Gaps in Language Model Generations, which reached macro-F1 0.89 on grounding acts. That number is on a different taxonomy and corpus, so it is not a target for Switchboard. Group-reweighted training is the standard fairness fix and is included.

All arms share data splits, label budgets, and a fixed hyperparameter search of equal size per arm. The key ablation is a zero-diversity ensemble, made of identical members differing only in dropout at inference. It isolates whether disagreement, rather than mere averaging or extra compute, causes any disparity change.

Beating these baselines would mean three things. First, calibrated ensembles reduce the disparity ratio and directional amplification below calibrated single and compute-matched models, with non-overlapping bootstrap intervals over at least ten seeds. Second, this happens at no loss of macro-F1. Third, group-level disagreement predicts the group-level improvement rate with a correlation comparable to the overall-accuracy relation reported in When are ensembles really effective?.

Why not just…?

Why not just calibrate thresholds per group? This is the real threat. If calibration erases amplification, as it nearly did in Directional Bias Amplification, ensembling has no fairness role here. The experiments settle it by reporting amplification after calibration for single models and ensembles separately. The decomposition into calibration-removable and residual parts is itself a result. If the residual near $\psi \approx 1$ is negligible, we will say so. The effective theory predicts it will not be negligible there, because its peak is noise- and variance-driven rather than threshold-driven.

Why not just use a bigger single model or better pretraining? An Effective Theory of Bias Amplification shows that more capacity lowers average error without reliably closing group gaps. The compute-matched arm tests whether that holds for dialogue encoders. Self-supervised pretraining is held fixed across arms, since it changes the representation rather than how predictions are combined. Label-efficiency work such as Self-supervised representation learning from 12-lead ECG data shows the largest gains on rare classes, which suggests pretraining and ensembling are complements.

Why not just prompt a frontier model and skip training? It is cheap and must be reported. But FairBelief finds harmful associations concentrated at peak likelihood, so a single prompted model's most confident act predictions are where speaker stereotypes would surface. A prompted model also offers no diversity to diagnose. We measure its group disparity on identical test splits.

Why not just reweight groups? Reweighting targets imbalance, but the theory predicts amplification even with balanced groups whose noise differs. The balanced sex split with injected noise is designed so that reweighting has nothing to correct. If reweighting still matches ensembles there, the variance account is wrong.

Proposed Experiments

The first decisive check tests the identity-plus-prediction. Across all probe ensembles, regress $\mathrm{EIR}_g$ on $\mathrm{DER}_g$ per group. Then test whether the predicted $\Delta_{\text{ens}}$, computed from unlabeled disagreement plus a small labeled estimate of $e_g$, matches the measured value.

The second check plots disparity and ensemble gain against $\psi$ for probes. Both should peak near $\psi = 1$, with ensembles flattening the disparity peak most there. In fine-tuned models, disagreement should collapse and the fairness benefit should vanish, mirroring the interpolation cliff.

The third check is the noise-injection experiment on balanced groups, which tests the noise mechanism directly. The fourth compares DANA-style rule arbitration against probability averaging. Rule arbitration discards the disagreement signal, so we expect it to lose the fairness gain even when its accuracy is similar.

A practical payoff follows if the first check succeeds. Groups with low unlabeled disagreement are ones ensembling cannot help, so they are where scarce labels should go. We test this as a label-allocation policy against uniform allocation at matched budgets.