Speech separation systems pull individual voices out of a recording where several people talk at once. They are scored by metrics that compare the output with a clean recording of the target speaker and count every bit of other speech as error. That accounting is already known to disagree with human listeners in studio conditions. This proposal argues it fails more badly, and in a predictable direction, in the setting separation is meant for: a crowded room where people stand in small conversational clusters. In such a room, whether a leaked voice matters depends on who that voice belongs to socially. It might be a partner in the same conversation, someone in a side conversation that split off within the same cluster, someone standing nearby who is not engaged, or a stranger in another group. Signal metrics treat all four cases identically, weighted by energy. We hypothesize that human listeners do not.
The idea combines two literatures that have not met. The first is F-formation analysis, the study of the circles people form when they converse. It gives a vocabulary and working detectors for exactly these social roles. The second is work on the gap between objective and perceptual audio metrics, which has produced perceptually grounded scores such as MAPSS. MAPSS separates "the target sounds damaged" from "other talkers leak in." It still treats all other talkers as equally unwanted. We propose to measure whether human quality judgments of separated speech depend on the social role of the leaked talker, holding the acoustics fixed. If they do, we will build a role-aware extension of perceptual leakage scoring and test whether it predicts those judgments better than signal metrics, MAPSS, and simple distance-based weighting.
Core Claim
In free-standing conversational groups, the perceptual cost of residual leakage in separated speech depends on the leaked talker's conversational relation to the target. The relevant relations are same floor, schismed sub-floor within the same F-formation, peripheral associate, or other group. (A conversational floor is the set of people actually sharing one conversation; a schism is when a group of four or more splits into two simultaneous conversations.) This dependence persists after controlling for distance and acoustic level. As a result, the divergence between signal metrics and human ratings widens precisely in socially complex configurations: schisms, associates, and adjacent groups. A metric that weights leakage by social role, estimated from body position and orientation, should recover much of the lost agreement. The claim is falsifiable in a clean way. If listeners rate identical audio the same regardless of the social role assigned to the leaked voice, the social layer is unnecessary and existing perceptual metrics suffice.
Where This Sits
The motivating evidence comes from both sides. On the social side, Towards automatic estimation of conversation floors within F-formations showed that groups of four or more on MatchNMingle routinely split into multiple floors. It detected these splits from 2-second simultaneous turns using only binary speaking flags and never touched audio. Beyond F-Formations defined in-group and out-group associates, people physically near a group but not fully in it. It also reported that annotators agree on associates only 16% of the time, compared with 94% for full members. This low agreement is a warning: social role labels are uncertain, and our design must use roles where they are reliable. On the audio side, MAPSS splits separation failure into Perceptual Match and Perceptual Separation using manifold distances over self-supervised speech features. It was validated only on studio mixtures in SEBASS. MARS-Sep showed that SDR-optimized separators keep semantically wrong but spectrally similar interference. Diffusion-based Signal Refiner reported that perceptual gaps persist even when objective scores improve.
The closest connecting work is Listen, Watch and Understand at the Cocktail Party. It uses face crops and linguistic context for separation on broadcast LRS2 clips and scores only SDR. It contains no body geometry, no groups, and no perceptual evaluation. SALSA supplies real badge audio, positions, and orientations from mingling scenes, but used the audio only for speaker detection. Perceived Conversation Quality collected observer ratings of how good conversations were, which is social quality rather than audio quality. No prior work in the file asks whether perceived separation quality depends on social structure. Our method contribution is reuse plus one addition. We reuse MAPSS's leakage scoring, and the F-formation detectors GCFF (F-Formation Detection) and Spatial-F, unchanged. What we add is a role-dependent weighting, together with the listening experiments that justify or refute it.
Research Program
The first component is a semi-synthetic benchmark built from real social geometry. Real mingling recordings lack clean per-speaker references, so SDR cannot even be computed on them. We therefore take the annotated trajectories from SALSA and MatchNMingle: 3D positions, head and body orientations (annotated every 3 seconds in SALSA), F-formation labels, and speaking-status timelines. These define where and when each simulated person talks. We fill those turns with clean read speech (LibriSpeech-style utterances) and render each scene acoustically. The rendering uses a reverberant room model, distance attenuation, and head directivity driven by the annotated orientations. Each render yields a mixture at a listener position plus clean references for every talker.
Because the geometry comes from real behavior, schisms, associates, and adjacent groups occur at their natural frequencies and layouts. Because the audio is rendered, we can make the decisive manipulation. We swap which rendered voice is labeled as belonging to which social role while holding its position, level, and timing fixed. We do this by reassigning floor membership in the speaking-status timeline. Social role then becomes an independent variable decoupled from acoustics.
The second component is a stimulus set with controlled and natural errors. Controlled stimuli inject residual leakage from one chosen talker at fixed signal-to-interference ratios into an otherwise clean target. Natural stimuli come from real separators: a mask-based audio-visual separator in the style of Li and Qian, a diffusion refiner in the style of Diffiner, and a semantically rewarded separator in the style of MARS-Sep, all run on the rendered mixtures.
The third component is a context-framed listening test. Raters see a top-down animated schematic of the scene, are told "you are standing as person A, talking with B and C," and rate the separated stream of the target in a MUSHRA protocol (a standard listening test with a hidden reference and anchors). The key within-stimulus manipulation shows the identical audio under two framings. In one framing the leaked voice is a same-floor partner; in the other it is an associate or another group's speaker. A context-free control condition presents the audio with no schematic.
The fourth component is a role-weighted leakage score. Let $\mathrm{PS}_j$ be MAPSS's Perceptual Separation computed between the output and competing talker $j$ alone, and let $r(j)$ be $j$'s social role relative to the target. We score
$$S = \mathrm{PM} + \sum_{j} w_{r(j)} \, \mathrm{PS}_j ,$$
where PM is MAPSS's Perceptual Match and $w_r$ is one weight per role category, five categories in total. The weights are fit on half the ratings and frozen. Roles are computed either from annotations (oracle) or from GCFF and Spatial-F groups combined with Raman and Hung's 2-second floor heuristic (estimated). Comparing the two settings tells us how much the score degrades under realistic pose error. SALSA reports head-pose errors of 42 to 49 degrees, so the gap could be large.
Baselines
The naive reference point is scale-invariant SDR and SIR computed against the target's clean reference. This is what Listen, Watch and Understand at the Cocktail Party reports, reaching 11.68 dB SDR on LRS2 with its full model. That number is from a different dataset and only anchors what that separator achieves. We also report PESQ and STOI on the same stimuli. All baselines score exactly the same stimuli against the same held-out human ratings.
The strongest existing competitor is unmodified MAPSS, which uses PM plus unweighted PS over all competing talkers. It is the only metric in the file built to separate leakage from distortion. On SEBASS it ranked first or second in Pearson and Spearman correlation against 18 metrics, but the file gives no scalar correlation. In-the-wild social audio has no reported result on Lacuna for MAPSS or any perceptual metric. We run MAPSS with its published layer choice and distortion bank. Our weighted variant uses the same features, so the only difference is the role weights. Beating it means a statistically significant improvement in Spearman correlation with held-out ratings, assessed by bootstrap over raters and scenes, and concentrated in schism and associate scenes rather than in simple dyads.
Two ablations isolate the key ingredient. In the first, the weights are a fitted function of physical distance between leaked talker and target. In the second, the weights are a fitted function of the leaked talker's received level. Both use the same number of free parameters as the role weights and are fit on the same split. If either matches the role weights, the social layer adds nothing beyond geometry.
A learned quality predictor fine-tuned on half our ratings serves as a capacity-matched upper reference. It follows SongEval's approach, where training on human ratings reached Pearson 0.912 versus 0.61 to 0.66 for Audiobox-Aesthetic in SongEval, on music aesthetics.
For the upstream social estimates, the reference detectors are GCFF and Spatial-F. GCFF reports a combined F1 of 0.89 across five datasets. Spatial-F reports F1 of 0.94 on Idiap Poster under strict criteria. For floor estimation, no reported accuracy number exists on Lacuna.
Why not just…?
Why not just weight leakage by distance or loudness instead of social role? This is the most serious threat. Same-floor partners tend to stand closer and sound louder than other groups. Real data cannot separate the two explanations, which is why the benchmark is semi-synthetic and the role-swap manipulation holds position and level fixed. The distance and level ablations are fit with equal parameter counts. If they match the role-weighted score on held-out ratings, we will report that social structure is unnecessary for evaluation and that geometry suffices.
Why not just use MAPSS as it is? Its Perceptual Separation term may already track what listeners care about, if listeners penalize all leakage alike. The context-free versus context-framed comparison settles this. If ratings of identical audio do not shift with framing, unweighted MAPSS is the right metric and our extension should be dropped. The question cannot be argued in advance; it is an empirical one.
Why not just train a quality predictor on the new ratings? A learned predictor may correlate well. It will not say why. It will also not transfer to scenes whose social layouts are absent from training. We include it as an upper reference and test transfer from SALSA-derived scenes to MatchNMingle-derived scenes. A five-parameter role-weighted score that transfers better would justify the structured approach. If the learned model transfers just as well, we report that honestly.
Why not just use speaking-status flags, as the floor-estimation work does? Binary flags can say who belongs to which floor. They say nothing about how damaged or contaminated a separated voice sounds, which is the quantity being evaluated. We use them as the estimator of social roles, not as a substitute for the metric.
Proposed Experiments
The first decisive check is the framing effect. Using controlled-leakage stimuli at three SIR levels across the five role categories, we test whether MUSHRA ratings of identical audio differ by assigned role, with a mixed-effects model with rater and scene random effects. The prediction is that associate and other-group leakage is rated worse than same-floor leakage at equal SIR. Schismed sub-floor leakage should fall in between, consistent with Raman and Hung's evidence that schisms are real but transient. A null result here ends the metric-building arm.
The second check measures the divergence. We compute the rank correlation between SDR and human ratings separately for dyads, single-floor groups, schismed groups, and scenes with associates or adjacent groups. The prediction is that correlation drops in the latter categories, and that it drops more than MAPSS's correlation does.
The third check is metric validity on natural separator outputs. The role-weighted score is compared with SDR, PESQ, STOI, unweighted MAPSS, distance and level weighting, and the learned predictor on held-out scenes. Comparisons are run within dataset and across datasets (SALSA-trained weights tested on MatchNMingle-derived scenes and vice versa).
The fourth check tests robustness to estimated roles. We replace oracle roles with GCFF or Spatial-F groups and heuristic floors, and report how much of the oracle gain survives. This exposes how upstream pose error limits the approach. Associate roles should be treated as low-confidence, given 16% annotator agreement, and we report results with and without that category.
The final check is ecological validity. On real SALSA badge audio, where no clean reference exists, we collect a smaller set of context-framed ratings of separator outputs. We then test whether the reference-free parts of the score keep the same ordering of separators. Here only the role-weighted leakage terms are computable from separated estimates, not from references, so this check is limited to ranking systems rather than scoring clips.
References
- Towards automatic estimation of conversation floors within F-formations
- Beyond F-Formations: Determining Social Involvement in Free Standing Conversing Groups from Static Images
- MAPSS: Manifold-based Assessment of Perceptual Source Separation
- MARS-Sep: Multimodal-Aligned Reinforced Sound Separation
- Diffusion-based Signal Refiner for Speech Enhancement and Separation
- Listen, Watch and Understand at the Cocktail Party: Audio-Visual-Contextual Speech Separation
- SALSA: A Novel Dataset for Multimodal Group Behavior Analysis
- Perceived Conversation Quality in Spontaneous Interactions
- F-Formation Detection: Individuating Free-Standing Conversational Groups in Images
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation