The Conversation Partner as a Room Microphone: Removing Room Acoustics from Speaker Embeddings for Small-Cohort Emotion Recognition

Emotion recognizers trained on small acted corpora such as IEMOCAP, which has only ten actors, already learn shortcuts. They tie a particular voice to a particular emotion label instead of learning how emotion sounds. Moving these systems into ordinary reverberant rooms adds a second shortcut. Reverberation is the smearing caused by sound bouncing off walls. It leaks into the speaker embedding, the fixed-length vector a speaker-recognition network produces to summarize who is talking. That vector then encodes the room along with the voice. With ten speakers, room, speaker and session are easily confounded, and a classifier can end up recognizing rooms. We propose a single-channel fix that uses a structural fact of conversation. In a dyadic recording, both talkers share the same room, so the other person's speech acts as a free acoustic reference for that room. We estimate the room's contribution from the part of both partners' embeddings that lies in a learned "room direction" of embedding space, subtract it, and pass the cleaned speaker vector to an emotion classifier that conditions on speaker identity.

The claim is that room leakage in speaker embeddings is a measurable and separate failure mode in small-cohort emotion recognition. It can be removed with no microphone array, no video, and no acoustic calibration sweep. Removing only the room part, and not the speaker as a whole, keeps the speaker information that helps interpret emotion. This also gives a direct test of an open disagreement in the field: should speaker identity be used to calibrate emotion judgments, or scrubbed out as a confound? Our position is that the disagreement is partly an artifact of clean audio. Under reverberation, what should be scrubbed is the room, not the speaker.

Where this sits

The disagreement comes from two IEMOCAP papers. Speaker Attentive Speech Emotion Recognition adds a speaker-identification branch whose outputs the emotion branch attends to. They report 77.22% unweighted average recall on four classes. Speaker Normalization for Self-supervised Speech Emotion Recognition does the opposite. It uses adversarial gradient reversal against a speaker classifier trained with VoxCeleb so that HuBERT features become speaker-agnostic. They report 74.2% weighted accuracy in leave-one-session-out evaluation. Both papers motivate this work. Both assume clean close-microphone audio, so neither can say what happens when the "speaker" signal is contaminated by the room.

The evidence that reverberation is severe comes from Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment. On RAVDESS with real ACE room impulse responses, single-channel audio accuracy falls from about 80% to roughly 40.5% in a lecture room with a 1220 ms reverberation time, even with RIR-augmented training. Accuracy recovers to 78.3% only with three microphones and video. That paper motivates this one and also constrains it: augmentation alone was not enough for them. They also studied isolated sentences, so they had no conversation partner to exploit.

Personalized Dereverberation of Speech shows that knowing the room helps a great deal. They reach PESQ 2.1 against 1.7 for WPE and 1.0 for Demucs. Their room knowledge comes from an active sine-sweep measurement plus five minutes of enrollment speech. Our proposal replaces the sweep with the interlocutor, and it works in embedding space rather than on the waveform.

The embedding extractors themselves supply the method. These are x-vector statistics pooling and ECAPA-TDNN. Transforming the Embeddings shows that such embeddings transfer to emotion recognition with a small 1.79M-parameter CNN, reaching 70.53% on CREMA-D. That parameter scale matters when there are ten speakers, since Lee & Tashev found that larger networks overfit IEMOCAP.

The small-cohort literature explains why we care about the speaker/room confound rather than raw accuracy. Bresch et al. show that individual anatomy can masquerade as a general pattern in small samples, and Pouplier et al. show the same thing with a two-participant study. We treat the room as another individual-level nuisance of this kind.

Core mechanism

Let $e_i$ be the ECAPA-TDNN embedding of utterance $i$ in a session. We model reverberation as an approximately additive shift inside a low-dimensional subspace:

$$e_i \approx s_i + R\,r_{\text{room}} + \epsilon_i .$$

Here $s_i$ is the embedding the utterance would have in a dry room. The matrix $R$, of size $d \times k$ with $k \ll d$, spans the room subspace. The vector $r_{\text{room}}$ gives the room's coordinates, and $\epsilon_i$ is residual noise.

We learn $R$ offline on external data. We take the same utterance dry and convolved with many simulated and measured RIRs, drawn from LibriSpeech and VoxCeleb with gpuRIR and diffuse simulation as in Improving Reverberant Speech Training Using Diffuse Acoustic Simulation. $R$ is the set of top principal directions of the reverberant-minus-dry differences, constrained to be orthogonal to the leading between-speaker directions. The additive-shift assumption is loose. It is plausible because statistics pooling averages frame features, and a convolutional channel shifts log-spectral frame features roughly additively. We test the assumption directly rather than relying on it.

At test time, a session's room is estimated from every utterance by both partners:

$$\hat r = R^{\top}\big(\bar e_{\text{session}} - m\big).$$

Here $\bar e_{\text{session}}$ averages both partners' embeddings and $m$ is the dry-speech mean. We then set $\hat s_i = e_i - R\hat r$. Pooling across two different voices is the key ingredient. Speaker-specific content that leaks into the room subspace partly cancels across partners, while the room component is shared by construction. The cleaned $\hat s_i$ then feeds speaker-attentive calibration in the style of Le Moine et al., built on a Phukan-scale lightweight classifier over log-mel features.

Research Program

The first part builds the benchmark the field lacks: reverberant IEMOCAP with controlled confounding. Each session's two speakers are placed at different source positions in a shared simulated room, or in a shared real room when using ACE RIRs. We define two protocols.

In the confounded protocol, each training session gets its own room family. At test time, rooms are swapped across sessions. A model that learned rooms instead of emotions shows a gap between matched-room and swapped-room test accuracy. We call this the room-swap gap.

In the unconfounded protocol, rooms are randomized per session and held out at test time. Evaluation stays leave-one-session-out, as in Gat et al., and we report unweighted and weighted accuracy.

The second part is diagnostic. On reverberant IEMOCAP we measure how well a linear probe can predict room identity from raw ECAPA embeddings, and speaker-verification equal error rate (EER). We also run the same probes on HuBERT/WavLM layers. This tests Gap 4 from the research file directly: do embeddings cluster by room rather than by vocal tract?

The third part runs the method and resolves the calibration-versus-removal question. We cross two choices of speaker signal (raw, room-cleaned, or adversarially removed) with calibration on or off. This shows which combination holds up under reverberation.

The fourth part is a real-acoustics check, since simulation may flatter the additive model. We re-record held-out IEMOCAP sessions by playing both talkers through two loudspeakers at different positions in two or three real rooms, captured with a single far-field microphone.

Baselines

The naive reference point trains the Phukan-style x-vector-plus-MFCC classifier on clean IEMOCAP and tests it on reverberant IEMOCAP with no treatment. Paired with the same model's clean score, it sets the size of the problem. No reported result was found on Lacuna for any system on reverberant IEMOCAP. The closest anchor is the roughly 40.5% single-channel figure on reverberant RAVDESS from Cohen et al. That figure is on a different corpus and is not directly comparable.

The strongest existing method is HuBERT Large fine-tuned with adversarial speaker normalization (Gat et al.), which reports 74.2% weighted accuracy on clean IEMOCAP in leave-one-session-out evaluation. We first reproduce it on clean data. A reproduction within about one point of 74.2% is the check that our pipeline is sound. We then train and test it on the same reverberant protocols. It gets RIR augmentation from the same RIR pool we use to learn $R$, and the same hyperparameter-search budget, measured in trials per fold, as our system.

This baseline is the real threat in the confounded protocol. When room is tied to session, adversarially removing speaker identity may remove room as a side effect. We expect its advantage to disappear in the unconfounded protocol, where room varies independently of speaker. Speaker-attentive SSA-CRNN (Le Moine et al., 77.22% unweighted recall on clean four-class IEMOCAP) is the second strong baseline. It is run under the same augmentation and budget.

Beating these baselines means two things. The first is higher unweighted accuracy in held-out real and simulated rooms, with a difference outside the fold-to-fold spread across the five sessions. The second is a room-swap gap no larger than the best competitor's.

Two ablations isolate the key ingredient. Session-mean normalization subtracts $\bar e_{\text{session}}$ in the full embedding space instead of only inside $R$. Single-speaker estimation computes $\hat r$ from one partner's turns only. If full-space session-mean normalization matches our method, the room subspace adds nothing. If single-speaker estimation matches it, the interlocutor adds nothing. We would report either outcome as a finding.

Why not just…?

Why not just train on RIR-augmented data? This is the first thing to try, and we include it in every baseline and in our system. Cohen et al. trained with RIR augmentation and still fell to about 40.5% at high reverberation. With ten speakers there is a further risk that augmentation creates speaker-room-label combinations a network can memorize. The comparison of augmentation alone against augmentation plus projection, under the unconfounded protocol, settles whether projection adds anything beyond augmentation.

Why not just dereverberate the waveform first with WPE or a neural enhancer? Xu et al. measured WPE at PESQ 1.7 and Demucs at 1.0 on real rooms. Both are optimized for signal quality rather than the pitch contour and spectral balance that emotion depends on; Busso et al. find that valence is already hard from spectral features alone (Busso et al.). We run DNN-WPE (cochlear-implant study) as a front end to both the baseline and our system. If WPE plus the plain classifier matches us, embedding-space cleanup is unnecessary. We also check whether combining the two helps.

Why not just fine-tune a large self-supervised model and let it absorb everything? Gat et al. show that plain HuBERT fine-tuning learns speaker shortcuts on IEMOCAP, and Lee & Tashev show that larger networks overfit it. The room probes on HuBERT layers will show whether those representations also carry room identity. The adversarial HuBERT baseline is exactly this alternative, run at its strongest.

Why not just subtract the session mean, like cepstral mean normalization? This is the most dangerous simple alternative, which is why it is a named ablation. Our argument against it is specific. In a mixed-gender dyad, the full-space session mean contains the average of the two voices. Subtracting it erases absolute speaker traits such as pitch register, which speaker-attentive calibration needs. Restricting the subtraction to the room subspace should keep them. The prediction is testable: speaker-verification EER and calibrated emotion accuracy should both be worse under session-mean normalization than under subspace projection. If they are not, we adopt the simpler method.

Decisive checks

The idea succeeds if three things hold together on held-out rooms, including the re-recorded real rooms. First, the room probe on cleaned embeddings falls close to chance while speaker-verification EER stays within a small margin of its clean-audio value. Second, the room-swap gap under the confounded protocol shrinks relative to both raw embeddings and the adversarial HuBERT baseline. Third, speaker calibration applied to cleaned embeddings beats both adversarial removal and calibration on raw embeddings in the unconfounded protocol.

The idea fails if any of three things happens. It fails if the additive-subspace fit breaks down on real rooms, which we would see as a residual room probe well above chance. It fails if session-mean normalization or WPE matches our method. It fails if interlocutor pooling gives no gain over single-speaker estimation. In that last case the "partner as room microphone" claim is wrong, even if subspace projection itself helps.

A secondary analysis reports results separately for arousal-heavy and valence-heavy class pairs. This tests whether room removal mainly recovers arousal cues, which reverberant smearing of the energy envelope would be expected to damage most, following the modulation-domain account in Zheng et al..