Teaching Microphone-Array Detectors What the Array Can Actually Hear

Systems that detect overlapping speech in meetings from a distant microphone array are trained on labels made from close-talk headsets and human segmentation. Those labels record what happened in the room. They do not record what the array could hear. When a second talker murmurs a backchannel at twenty decibels below the main speaker, from across a reverberant room, the label still says "overlap = 1" with full certainty. The detector is then pushed to be fully confident about an event that may be physically buried at the array. We propose replacing these hard targets with observability-weighted soft targets: per-frame probabilities computed from the parallel close-talk recordings that already ship with meeting corpora. Each target is shrunk toward zero exactly when the weaker talker is unlikely to be audible at a distance. The detector front-end we study is a self-attention channel combiner, a network that treats each microphone as a token and learns, frame by frame, how much weight to give each channel. The soft targets are free, they vary by instance, and they come from information the model never sees at test time.

The bet is that this matters most for channel-attention models specifically, for three reasons. Their attention weights act as learned spatial filters and could, in principle, steer toward a faint talker. Hard labels give them a confident but often unsatisfiable instruction to do so. And the channel-dropout training these models use for robustness feeds them weaker evidence while keeping the target unchanged. We expect soft targets that track physical audibility to give better-calibrated overlap probabilities, more stable training of the complex-valued combiners, and better robustness when microphones are removed. A cheap ablation that keeps the softness but scrambles which frames receive it will tell us whether the information or the smoothing is doing the work.

Core Claim

Hard overlap labels contain two kinds of error relative to a distant array. The first is temporal: the exact onset and offset are fuzzy. The second, and in our view larger, is acoustic observability: some labelled overlap frames carry almost no evidence at the array. Temporal error is what generic smoothing addresses. Observability error is instance-specific, depends on the relative levels of the talkers, and can be measured from the close-talk channels.

We define the soft overlap target for frame $t$ as

$$\tilde y_t = y_t \cdot \sigma\!\left(\frac{r_t - \tau}{s}\right),$$

where $y_t \in \{0,1\}$ is the original hard label, $r_t$ is the level of the second-strongest active talker relative to the strongest (in decibels, computed from close-talk headset energy within the annotated segments), $\sigma$ is the logistic function, and $\tau, s$ set the audibility midpoint and softness. Because of the multiplication, the target can only lower confidence on labelled overlap frames. It never invents overlap where annotators marked none. In the language of the perspectivist argument in Toward a Perspectivist Turn in Ground Truthing for Predictive Computing, the label is treated as correct about the room and uncertain about the sensor.

The claim is that a channel-attention detector trained against $\tilde y$ will be better calibrated at temperature one and no worse in thresholded F1. We also expect it to degrade more gracefully when channels are dropped. The second and third effects should be larger for complex-valued combiners than for magnitude-only ones.

Where This Sits

Methods we reuse. The channel-combination family comes from Mariotte and colleagues. Microphone Array Channel Combination Algorithms for Overlapped Speech Detection introduced the Self-Attention Channel Combinator (SACC) and its complex extension cSACC on AMI. Channel-Combination Algorithms for Robust Distant Voice Activity and Overlapped Speech Detection added analytic and explicit/implicit complex variants (SACC+A, EcSACC, IcSACC) and a channel-number invariant loss. ASoBO: Attentive Beamformer Selection for Distant Speaker Diarization in Meetings applied the same attention over a bank of fixed beamformers. All of these train on hard binary frame targets. We keep their architectures unchanged and replace only the supervision.

Work that motivates the proposal. Reducing Supervision Uncertainty Induces Model Miscalibration showed on subjective text datasets that collapsing rater distributions into hard labels makes models overconfident. In that work, soft-trained models were best calibrated at $T=1$, and post-hoc temperature scaling could not recover the instance-level information that had been discarded. Calibrating Histopathology Image Classifiers using Label Smoothing showed that scaling smoothing by per-image pathologist disagreement cut calibration error from 8.9% to 2.8%. It also showed that a two-pass, confidence-based proxy reached 3.2% when there was only one annotator. A Case for Soft Loss Functions found that soft crowd labels can even raise accuracy (66.64% vs 65.22% on CIFAR-10H).

Where we depart from that literature. All of this soft-label work draws its distributions from human raters or from the model's own confidence, and none of it touches multi-sensor inputs. The closest audio work, End-to-End Label Uncertainty Modeling in Speech Emotion Recognition, models rater disagreement on single-channel emotion labels. Our soft targets come from neither raters nor the model. They come from a privileged physical measurement, the close-talk headsets, about the gap between what happened and what the test-time sensor can observe.

A constraint we take seriously. The perspectivist critique and the evaluation discussion in Toward a Perspectivist Turn in Ground Truthing for Predictive Computing warn that hard F1 penalizes honest uncertainty on disputed items. Our soft model will deliberately be less confident on quiet overlaps that the hard reference still counts as positives. We therefore report F1 stratified by $r_t$ and treat a small F1 loss confined to the inaudible stratum as expected rather than as failure.

Research Program

The first component is target construction on AMI. AMI provides per-speaker headset channels in parallel with the 8-microphone circular Array 1 and the 4-microphone Array 2. For each annotated overlap frame we compute $r_t$ from headset energies, restricted to talkers whose manual segments are active. Restricting to annotated talkers means crosstalk from neighbouring headsets cannot create spurious speakers. It can only bias levels, and we check that bias by measuring $r_t$ on single-speaker regions, where it should sit far below $\tau$. The audibility parameters $\tau$ and $s$ are chosen on the development set by maximizing log-likelihood of a hard-trained model's array-side predictions. This gives an empirical audibility curve rather than a hand-picked one. We also report sensitivity across a grid.

The second component is subset-aware targets in simulation. Using LibriSpeech convolved with simulated room impulse responses, as in the simulation protocols of the SACC and SCA-TSVAD papers, the per-microphone image of every source is known exactly. Here we compute $r_t$ not from close-talk levels but from the best per-source signal-to-interference ratio available among the microphones actually present in the training sample. When channel-dropout training removes the microphone nearest the quiet talker, the target drops accordingly. This directly addresses an inconsistency in channel-number invariant training (Channel-Combination Algorithms for Robust Distant Voice Activity and Overlapped Speech Detection): it removes evidence while holding the target fixed. Subset-aware targets cannot be computed on real AMI, so simulation is where we test whether target–evidence consistency, rather than softness alone, drives robustness.

The third component is analysis of the attention itself. Following the time-averaged beampattern analysis of the cSACC paper, we measure on simulated rooms with known talker positions how much attention mass the combiner places toward the weaker talker's direction during overlap frames, as a function of $r_t$. We expect two different behaviours. A hard-trained complex combiner should show smeared, unstable patterns at low $r_t$, because it is chasing unreachable targets. A soft-trained one should steer toward the faint talker only when it is audible. We also track gradient-norm variance on the complex attention parameters. This tests the hypothesis behind the cSACC-versus-SACC gap reported in Microphone Array Channel Combination Algorithms for Overlapped Speech Detection (23% vs 19% gap reduction): that part of the complex models' difficulty comes from supervision rather than from complex parameterization per se.

Voice activity detection uses the same construction, with a speaker's level relative to the headset noise floor in place of $r_t$. It is a secondary target, because single-talker activity is usually audible at the array and the expected effect is smaller.

Baselines

The naive reference point is a single distant microphone with a temporal convolutional network (TCN) detector trained on hard labels. This is the floor against which the channel-combination papers measure their gap closure, and it tells us how much of any calibration gain is specific to multichannel attention.

The strongest existing method is EcSACC with a TCN back-end trained with hard multi-task cross-entropy, as in Channel-Combination Algorithms for Robust Distant Voice Activity and Overlapped Speech Detection. That paper reports an OSD F1-score of 68.4% on the AMI evaluation set, and a DER of 23.30% with a JER of 30.90% when used for diarization. No reported calibration result (ECE or Brier score) was found on Lacuna for any channel-attention detector on AMI, so we will establish one. We retrain EcSACC, IcSACC, cSACC and magnitude SACC+STFT ourselves under one recipe: same data splits, same channel-dropout schedule, same number of epochs, and the same hyperparameter search budget for every supervision variant. We first confirm that our hard-label EcSACC reproduces an F1 near 68.4%. Beating it means three things together: lower ECE and Brier score at $T=1$ than the hard model achieves even after its best post-hoc temperature; no drop in aggregate F1 beyond seed variance; and lower DER under overlap assignment.

The calibration competitor is the same hard-trained EcSACC with a single temperature fitted on the development set. Beating it requires better Brier score in the low-$r_t$ stratum, which is exactly where Reducing Supervision Uncertainty Induces Model Miscalibration predicts global scaling should fail.

Two generic-softening baselines are tuned with the same search budget as our $\tau, s$. One is uniform label smoothing over all frames. The other is temporal boundary smoothing, which blurs RTTM onsets and offsets over a collar-sized window.

Two baselines use the model's or a teacher's own confidence. The first is the confidence-aware two-pass smoothing of Calibrating Histopathology Image Classifiers using Label Smoothing, with a first-pass array model's confidence setting each frame's smoothing. The second is distillation from a detector trained on close-talk headset audio.

The ablation that isolates the key ingredient is shuffled observability. Within each recording, we randomly permute the $\tilde y$ values among labelled overlap frames. This keeps the exact amount and distribution of softness but destroys its link to acoustics. If shuffled targets match ours, our claim fails.

Why not just…?

Why not just temperature-scale the hard-trained detector? Temperature scaling multiplies every logit by the same factor. It therefore cannot make the model confident on a loud overlap and unconfident on a buried one at the same time. The text-domain evidence in Reducing Supervision Uncertainty Induces Model Miscalibration shows that hard-trained models remained worse in Brier score even after their best temperature. Audio could differ, though: if overconfidence on AMI happens to be roughly uniform across $r_t$, temperature scaling would be nearly sufficient. The $r_t$-stratified reliability diagrams settle this directly. If the hard model's miscalibration is flat across strata, we will say so, and the calibration part of our claim collapses to the robustness and optimization parts.

Why not just smooth the label boundaries in time? Temporal smoothing addresses onset uncertainty, which is real, but it leaves the interior of a quiet backchannel at full confidence. Our targets do the opposite. The two are complementary, and we run their combination. The threat is that boundary error dominates on AMI. In that case boundary smoothing alone would match us, and the per-stratum analysis would show our gains concentrated near segment edges rather than at low $r_t$. We would report that outcome.

Why not just use the model's own confidence, or a close-talk teacher, as the soft target? The confidence-aware proxy from Calibrating Histopathology Image Classifiers using Label Smoothing came close to true rater agreement in histopathology (3.2% vs 2.8% ECE), so it is a genuine threat. However, a first-pass array model's uncertainty mixes physical inaudibility with its own fitting errors and spatial confusions, and it reinforces whatever the first model got wrong. A close-talk teacher has the opposite problem: overlap is usually obvious in headset audio, so it produces nearly hard targets that again say nothing about the distant array. Both baselines are run at equal budget. If confidence-aware smoothing matches observability targets, the privileged-measurement argument is not needed and we would recommend the simpler method.

Why not just apply this to a single-microphone detector, or to channel averaging? If the effect is purely about labels, it should appear equally for any front-end. Our hypothesis is that channel attention benefits more, because its combination weights are what get pushed toward unreachable spatial solutions. We test this by measuring the soft-minus-hard improvement for the single-microphone baseline, for channel summing, and for each SACC variant. A uniform improvement across front-ends would mean the multichannel framing is incidental. We would then report the result as a labelling finding rather than an architecture finding.

Proposed Experiments

Data and arrays. The main experiments train and evaluate on AMI Array 1 with the standard split, and test cross-configuration on Array 2. On Array 2, invariant training is reported to improve SACC+STFT OSD F1 by 10.2%, which makes it a sensitive probe of robustness. AISHELL-4 serves as an out-of-domain evaluation for the ASoBO-style beamformer-attention variant. It has no headsets, so it supports evaluation only, not soft-target training. Simulated LibriSpeech rooms with random-subset channel dropout host the subset-aware targets and the beampattern analysis.

Metrics. We report OSD and VAD F1 and average precision, both overall and stratified into $r_t$ bins. We report ECE and Brier score at $T=1$ and after fitted temperature. For downstream diarization we report DER and JER with overlap assignment. For the attention analysis we report the attention mass toward the weaker talker's true direction and the variance of gradients on the complex parameters.

Decisive checks. The proposal stands or falls on four results. First, observability targets must beat shuffled observability in Brier score and low-stratum calibration; otherwise the information content is not what matters. Second, they must beat the best-tuned temperature-scaled hard EcSACC on low-$r_t$ Brier score while holding aggregate F1 within seed noise of the reproduced 68.4%. Third, in simulation, subset-aware targets must beat subset-agnostic soft targets under heavy channel dropout; otherwise target–evidence consistency is not the operative mechanism. Fourth, the soft-minus-hard improvement must be larger for complex combiners than for magnitude SACC. If that holds, it would be the first evidence that the cSACC performance gap is partly a supervision problem. If it does not, it cleanly rules that explanation out.