In tonal languages such as Mandarin and Thai, a syllable's meaning depends on its pitch contour. When the audio is noisy, or when a listener uses a cochlear implant that strips out pitch, the talker's face (head, eyebrow and lip movements) can partly stand in for the missing pitch. Human studies show this visual boost grows as audio gets worse. Machine lip-reading systems also gain from modelling tone explicitly. Nobody has asked whether that visual boost is delivered evenly across talkers. The traits that plausibly change it vary smoothly rather than in categories: skin lightness affects contrast around the lips and brows, voice register (a talker's typical pitch height) affects the audio side, and the size of a talker's facial movements differs from person to person. This proposal measures tone-recognition error, and specifically the gain from adding video, as a smooth function of these continuous talker traits. It then trains the recognizer so that error does not depend on appearance once we account for how much tone information the face actually carries.
The key distinction is between two reasons a talker might get less visual help. The first is that the face really shows less, because the talker moves less. That is an information limit, and no fairness constraint should hide it. The second is that the system reads the same movements worse on some faces, for example darker or older ones. That is a bias, and it can be fixed. We separate the two by conditioning on measured facial kinematics. We then enforce a kernel-smoothed "equal loss across a continuous attribute" penalty only within kinematic strata. This keeps the continuous-fairness method from simply flattening real differences in how much information talkers provide.
Core Claim
Our claim is that audio-visual tone recognizers distribute their visual benefit unevenly along continuous talker attributes, and that the unevenness is largest exactly where the visual stream matters most: at low signal-to-noise ratio (SNR) and under cochlear-implant simulation. A loss-parity regularizer, conditioned on noise level and facial-movement magnitude, can remove the appearance-driven part of this disparity at small cost in tone error rate. It does better than binning, pitch normalization or augmentation. The audit is informative even if the claim fails: flat error-versus-attribute curves would be a useful negative result for assistive audio-visual systems.
Grounding in Prior Work
Several papers motivate the gap by showing the visual tone signal is real but fragile. Visual cues in Mandarin tone perception found that video adds 1.3% tone identification at −3 dB babble and 15.3% at −12 dB. Multimodal perception of Mandarin tone for cochlear implant users found visual facilitation under cochlear-implant simulation that grew with degradation. So the visual benefit concentrates on the people with the worst audio, and any disparity in it lands on them.
Two further papers constrain the design. Different facial cues for different speech styles in Mandarin tone articulation showed that facial tone cues are mostly generic emphasis, "faster and further" movement in clear speech, rather than tone-specific mirroring of pitch. Visual tone information therefore scales with how vigorously a talker moves, which is why we must condition on movement magnitude rather than treat all error differences as bias. Robustness of tonal and segmental information in noise - auditory and visual contributions found zero visual gain for tone in open-set human identification. This warns that closed-set tone tasks may overstate how much the face helps, so we report both closed-set monosyllable and sentence-level settings.
On the machine side, A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading (CSSMCM) predicts pinyin, then tone, then characters from silent video. It reaches a 10.95% tone error rate (TER) and 32.48% character error rate on CMLR, but evaluates on 11 news anchors with no per-talker analysis. LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading reports persistent tone errors when reconstructing Mandarin speech from video. CogCM: Cognition-Inspired Contextual Modeling for Audio-Visual Speech Enhancement conditions enhancement on visual appearance features and reports large gains at −15 dB. None of the three measures who receives those gains. We reuse the CSSMCM tone head as our backbone architecture and add the audit and the constraint.
The fairness methods come from work on continuous sensitive attributes. Generalized Demographic Parity for Group Fairness measures how far the average prediction at attribute value $a$ drifts from the global average, estimated with a smoothing kernel. It shows the kernel estimator converges faster than binning, at $O(N^{-4/5})$ versus $O(N^{-2/3})$. Fairness-Aware Learning for Continuous Attributes and Treatments penalizes a density-estimated bound on maximal (nonlinear) correlation between predictions and the attribute. Fairness-Aware Grouping for Continuous Sensitive Variables: Application for Debiasing Face Analysis with respect to Skin Tone (FairGroups) showed that Fitzpatrick bins and a lightness cutoff hide localized bias in static face classification, and removed much of it with optimal-transport post-processing. All three were built for tabular data or single images, and all three target prediction independence.
For tone, the label is roughly independent of the talker's skin or register, so prediction parity is nearly automatic and misses the real harm, which is unequal error. We therefore move the kernel-smoothing machinery from predictions to per-sample loss. We also add conditioning on noise level and kinematics, which none of these papers consider.
Research Program
The program has three parts: a controlled audit, a conditional fairness objective, and a test on real held-out talkers.
Controlled audit. Existing Mandarin audio-visual data has few talkers: CMLR has 11 anchors, and the Garg et al. kinematics corpus has 20 speakers producing tone-labelled monosyllables in plain and clear styles. Few talkers cannot support smooth error curves across people, so the audit starts with within-talker counterfactual sweeps. We shift each talker's pitch register while preserving the contour shape and formants, using pitch-synchronous resynthesis. We shift skin lightness $L^*$ in CIELAB space, a color space with a perceptual lightness axis, on the video frames. We cross both sweeps with babble noise from 0 to −12 dB SNR and with cochlear-implant simulation.
For each condition we plot tone error and visual gain (audio-visual error minus audio-only error) against the continuous attribute. The sweeps are diagnostic only. Because we can train on the same transformations, they cannot serve as the final test.
Conditional fairness objective. Let $\ell_\theta$ be the per-clip tone loss of the recognizer with parameters $\theta$. Let $A$ be a continuous talker attribute (skin lightness $L^*$, speaker mean $f_0$, or age). Let $S$ be the noise condition and $K$ a stratum of measured facial-movement magnitude, from peak displacement and velocity as in Garg et al. The penalty is
$$ D(\theta)=\sum_{s,k}\int \Big|\,\hat{\mathbb E}[\ell_\theta \mid A=a,S=s,K=k]-\hat{\mathbb E}[\ell_\theta \mid S=s,K=k]\Big|\,dP_A(a \mid s,k), $$
where $\hat{\mathbb E}$ is a Gaussian-kernel smoother over $a$, following the kernel GDP estimator. The training objective adds $\lambda D(\theta)$ to cross-entropy, and sweeping $\lambda$ traces an error–disparity frontier.
Conditioning on $K$ is the key ingredient. It asks for equal loss among talkers who move equally, so it penalizes appearance-driven failure but not genuine information scarcity. Kernel estimates are noisy on mini-batches, as the fairness literature itself notes. We therefore keep a memory buffer of recent per-sample losses with their attributes and compute $D$ over the buffer. Because the conditioning space is low-dimensional (one attribute, one noise level, a few kinematic strata), density estimation remains feasible.
Real-talker test. The decisive evaluation uses real talkers never seen in training, whose attributes vary naturally rather than by counterfactual editing. We train on CMLR anchors plus part of the 20-speaker Garg corpus, and test with leave-speakers-out splits. CMLR and the Garg corpus alone give too narrow a spread of age, register and skin tone. We therefore also record a modest evaluation set of Mandarin talkers who span these ranges, reading the Mixdorff et al. monosyllable lists and short sentences. This lets closed-set and open-set results be compared with the human data.
Baselines
The naive reference point is an unconstrained audio-visual tone classifier. It uses a CSSMCM-style tone head fed by a 3D-CNN visual front-end and an audio encoder, trained with cross-entropy only. For silent-video tone recognition on CMLR, the strongest reported number is CSSMCM's TER of 10.95% (A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading). Our reimplementation must land near that value before any fairness comparison is meaningful. For tone error disparity along any continuous talker attribute, no reported result was found on Lacuna, so the disparity numbers for all methods will be new.
The strongest existing fairness competitor is the maximal-correlation penalty of Fairness-Aware Learning for Continuous Attributes and Treatments, applied to the recognizer's tone posteriors. We also test the post-hoc alternative, FairGroups partitioning plus optimal-transport correction from Fairness-Aware Grouping for Continuous Sensitive Variables. On CelebA that method reduced maximal-correlation bias from 0.126 to 0.039 while accuracy moved from 0.793 to 0.781. That result shows it works on static faces; it has not been demonstrated on video tone recognition.
Two further baselines are simpler practices that could remove disparities without any fairness math: per-utterance pitch normalization of the audio stream, and training-time augmentation over register and lightness. Two ablations isolate our ingredients. The first uses the same loss-parity penalty without kinematic conditioning. The second applies the penalty through the audio branch only versus the visual branch only.
To keep comparisons fair, all methods share the backbone, training data, augmentation pool (where applicable), epoch budget, and an equal hyperparameter search of about twenty runs over the fairness weight, kernel bandwidth or bin count. Everything is evaluated on the same held-out real talkers, in each noise condition.
We will count the method as beating a competitor under one condition. At matched TER, within 0.5 points, it must show lower kernel-estimated loss disparity along skin lightness and age at −9 and −12 dB and under cochlear-implant simulation. It must do this without lowering the error gap between high- and low-movement talkers that the unconstrained model shows, since that gap reflects genuine information differences.
Why not just…?
Why not just bin the attributes, for example into Fitzpatrick types or low/mid/high pitch, and apply standard group fairness? FairGroups showed that Fitzpatrick bins captured less inter-group fairness variance than learned partitions of continuous lightness (0.078 versus 0.092). It also showed that a single lightness cutoff can report near-zero disparity while splitting a strongly disadvantaged subgroup in half. Kernel GDP further converges faster than the histogram estimator. With only tens of talkers, bins will be sparse and their edges arbitrary. We run binned group equalization at matched tuning budget. If it matches our disparity curves, that is a real finding, and we would report it as evidence that talker counts in this domain are too small for the continuous machinery to matter.
Why not just normalize pitch per utterance, as speech researchers already do for tone? Normalization plausibly removes register effects in clean audio, and we expect it to be a strong competitor on the audio side. It has two limits. It does nothing for the visual stream. And under heavy babble or cochlear-implant simulation, the pitch estimate it relies on is itself unreliable. The cochlear-implant study found audio discrimination collapsed under simulation ($F(1,92)=1771.6$), which is exactly the regime in which the system leans on the face. The audit will show whether any register disparity survives normalization. If none does, register drops out as an attribute and the work focuses on appearance.
Why not just augment training with pitch shifts and lightness jitter? This is the most serious threat. Augmentation will flatten the counterfactual sweeps almost by construction, which is why those sweeps are not the final test. It does not obviously fix disparities driven by real correlates of age or skin tone that a color shift fails to imitate, such as texture, facial hair, wrinkles or camera exposure. The decisive comparison is therefore on real held-out talkers. If augmentation matches the conditional penalty there, we will say that the simple practice suffices.
Why not just use adversarial debiasing on the visual features? Adversarial training asks the representation to hide the attribute. That can erase useful signal: movement magnitude correlates with age, and the Garg et al. results make movement the main carrier of visual tone information. Adversarial training is also slower and less stable. Kernel GDP was reported to cut training time by more than 50% and to give over 1% higher accuracy at equal fairness on Adult (Generalized Demographic Parity for Group Fairness). We include an adversarial run only as a secondary check.
Proposed Experiments and Decisive Checks
We run three experiments. The first establishes the backbone near the CMLR 10.95% TER reference and produces counterfactual audit curves across register, lightness, noise level and cochlear-implant simulation. This tells us whether the visual gain falls with lightness or age, and whether that dependence grows as audio worsens.
The second is the real-talker comparison of all baselines on the held-out talker set, reporting TER, visual gain and kernel-estimated loss disparity with bootstrap intervals over talkers. The third compares machine visual gain with human visual gain on the Mixdorff monosyllable lists, in both closed-set and open-set formats. It tests whether equalizing the machine's visual benefit keeps it consistent with the human finding that open-set tone gets little visual help.
The idea is falsified if the unconstrained model's error curves are flat along appearance attributes once movement magnitude is controlled. It is also falsified if augmentation or pitch normalization matches the conditional penalty on real held-out talkers. The idea is supported if appearance-driven disparity appears mainly at low SNR and under cochlear-implant simulation, and if the conditional penalty removes it while leaving the movement-driven gap intact.
References
- Visual cues in Mandarin tone perception
- Multimodal perception of Mandarin tone for cochlear implant users
- Different facial cues for different speech styles in Mandarin tone articulation
- Robustness of tonal and segmental information in noise - auditory and visual contributions
- A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading
- LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
- CogCM: Cognition-Inspired Contextual Modeling for Audio-Visual Speech Enhancement
- Generalized Demographic Parity for Group Fairness
- Fairness-Aware Learning for Continuous Attributes and Treatments
- Fairness-Aware Grouping for Continuous Sensitive Variables: Application for Debiasing Face Analysis with respect to Skin Tone