Speech systems that fuse hand-crafted acoustic measurements with learned neural embeddings usually combine them in the crudest way available. Hand-crafted measurements include pitch, loudness, voicing and jitter, the frame-to-frame instability of pitch. The learned embeddings come from self-supervised models such as WavLM, which are trained on large amounts of unlabeled audio. Current systems either glue the two vectors together or average the scores of two separate classifiers. This note proposes a different division of labor. Inside a graph over the frames of an utterance, the physical measurements decide which frames should exchange information, and the neural embeddings supply what information is exchanged.
The core expectation is about disordered, accented or noisy speech. There, the geometry of self-supervised embedding space becomes unreliable, and frames that "look similar" to the network are often not acoustically comparable. Pitch, energy and voicing still reliably group, for example, all the sustained voiced frames of a speaker's modal register. Neighbor selection keyed on physics should therefore degrade less under domain shift than neighbor selection keyed on the embeddings themselves. This matters most for clinical voice biomarkers, where the training data are small and the test population is, by construction, acoustically out of distribution.
Core Claim
Dynamic neighbor weighting means computing, for each node in a graph, content-dependent weights over its neighbors instead of fixed structural averages. Graph Attention Networks introduced it. In the standard formulation, one node's features produce both the attention weights and the aggregated messages.
We split these two roles across the two feature families. Let $s_j$ be the self-supervised embedding of frame $j$. Let $p_j$ be a low-dimensional vector of frame-level physical descriptors: fundamental frequency, energy, zero-crossing rate, voicing probability, and local jitter and shimmer. The update for frame $i$ is
$$\alpha_{ij} = \frac{\exp\big(q(p_i)^\top k(p_j)\big)}{\sum_{m \in \mathcal{N}_i} \exp\big(q(p_i)^\top k(p_m)\big)}, \qquad h_i = s_i + \sum_{j \in \mathcal{N}_i} \alpha_{ij} W s_j,$$
The symbols are as follows. The functions $q$ and $k$ are small learned maps applied only to physical descriptors. $W$ is a learned projection of the neural embeddings. $\mathcal{N}_i$ is the set of frames that frame $i$ listens to.
The neighborhood itself is also chosen by physics rather than by a fixed count. Frame $j$ enters $\mathcal{N}_i$ when its physical affinity score $q(p_i)^\top k(p_j)$ exceeds a threshold $\tau(p_i)$ predicted from frame $i$'s own descriptors. As a result, a sustained vowel can gather many neighbors while a stop burst gathers few.
The claim is specific and falsifiable. At matched parameter count, this physics-keyed model should beat three alternatives, with the margin growing as train–test acoustic mismatch grows. The first alternative is the same graph with attention computed from the embeddings. The second computes attention from the concatenation of both. The third is concatenation-based fusion with no graph.
How This Differs from Existing Work
Three prior lines motivate the proposal.
Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation found that hybrid recognizers shift reliance toward spectral features when self-supervised features face domain mismatch. That is evidence that the physical view stays usable when the neural view does not. However, the paper exploits this only implicitly, through learned fusion weights. Anne Rowling Neurological Speech Corpus benchmarked eGeMAPS and deep features (UniSpeech, TitaNet) separately with random forests on neurodegenerative speech, reaching 70–80% accuracy. It never combined them, which leaves open whether a relational combination helps. Hybrid Expert Knowledge and Self-Supervised Learning for Diagnostic Modeling of Adductor Spasmodic and Primary Myotonic Dysphonia found that a hand-crafted branch alone reaches 81.8% against 83.3% for the WavLM hybrid. That is a warning that physical features already carry most of the clinical signal. It also means our gains must come from how they are used, not from their mere presence.
The closest methods fall into two groups. The first builds graphs over feature types. Graph-Based Multi-Feature Fusion Method for Speech Emotion Recognition treats five utterance-level feature streams (eGeMAPS, MFCC, two bag-of-audio-words histograms, and VGG Deep Spectrum) as five graph nodes. It learns multi-dimensional edges with cross-attention and aggregates with a convolutional graph network. Our graph instead has frames as nodes and uses physics only to route. With five nodes, "graph attention over feature types" collapses into learned gating, and we deliberately do not propose it.
The second group builds graphs over frames or patches. Graph Attention for Automated Audio Captioning builds attention graphs over pretrained audio network frames with a fixed top-25 mask. ATGNN: Audio Tagging Graph Neural Network builds k-nearest-neighbor graphs over spectrogram patches in learned feature space. Both choose neighbors from the same learned features they aggregate, and both fix the neighborhood size. We replace both choices.
Finally, Exploring Multilingual Unseen Speaker Emotion Recognition lets spectral features guide co-attention over Whisper embeddings, and reports a 10% drop when co-attention is removed. This is the nearest conceptual relative: physical cues steering attention over neural features. That work, however, uses pairwise cross-attention between two streams with learned spectral encoders. It includes neither interpretable frame-level descriptors, nor adaptive neighborhoods, nor a controlled test of domain-shift robustness. We isolate exactly the question it leaves implicit: whether who attends to whom should be decided by physics.
Research Program
We will build the model on frozen WavLM Base+ layer-weighted features (768 dimensions at 50 frames per second). A lightweight adapter follows Du et al.'s setup, so that the clinical comparison is like-for-like. Physical descriptors come from openSMILE's frame-level low-level descriptors, aligned to the 20 ms WavLM frame rate. They are standardized per speaker only where the benchmark protocol allows it. Two graph layers precede attentive pooling and a task head. This keeps depth shallow, given the over-smoothing that ATGNN reports for deeper graph stacks.
The threshold $\tau(p_i)$ is trained with a straight-through relaxation. It carries a mild penalty on expected neighborhood size, so that the model cannot trivially attend to everything.
The analysis has two parts beyond accuracy. First, we test whether learned neighborhoods track acoustic structure. We measure neighborhood size against voicing and local pitch stationarity, and the fraction of attention mass that stays within voiced regions. Second, we test the robustness mechanism directly. We corrupt the neural view alone by feeding WavLM noise-degraded audio while computing the physical descriptors from clean audio, and the reverse. We then check whether physics-keyed routing is insensitive to neural corruption, as claimed.
Baselines
The naive reference point is a WavLM-only model with mean pooling and the same head. It shows how much any fusion buys. A second reference is the hand-crafted-only model, following the CNN-EK branch of Du et al.
The strongest existing fusion method is concatenation into a 1D convolutional network, as in ComFeAT. On E-DAIC depression severity regression it reports RMSE 5.28 for TRILLsson plus LFCC, against 5.83 for TRILLsson alone and a prior best of 8.19. We will rerun ComFeAT with both its TRILLsson features and our WavLM features. This matters because simple concatenation is exactly what our method claims to beat. Beating it means lower RMSE than our own faithful reproduction under identical splits, and landing below 5.28 is the external sanity target. On the dysphonia corpus the strongest reported result is the WavLM-adapter late-fusion hybrid at 83.3% accuracy (Du et al.), which we reproduce as late fusion. On the neurodegenerative corpus the reference is Tam et al.'s 70–80% accuracy and $R^2$ of 0.40–0.48 on the speech sub-score of the ALS Functional Rating Scale (ALSFRS-R). We report our numbers under their speaker-grouped five-fold protocol.
For graph baselines we run the fixed top-25 frame attention of GraphAC on WavLM frames. No reported result was found on Lacuna for GraphAC on speech classification. We also run a standard transformer layer over concatenated physical and neural frames, which is the fully connected graph with attention computed from everything.
The ablation that isolates the key ingredient keeps the architecture fixed and changes only what feeds $q$ and $k$: physical descriptors, WavLM embeddings, or their concatenation. A second ablation fixes the key source but swaps the adaptive threshold for a fixed top-k.
All systems share frozen encoders, data splits, a matched parameter budget within 10%, and the same hyperparameter search budget of 30 trials each on validation data. Results are averaged over five seeds, with paired bootstrap confidence intervals.
Why not just…?
Why not just concatenate physical features onto each frame and let a transformer sort it out? This is the most serious threat. A transformer given $[p_j; s_j]$ could in principle learn to key on the physical dimensions alone. We expect it will not do so reliably with a few hundred training speakers, because the 768 neural dimensions dominate the dot products. This mirrors the dimensional imbalance the literature notes for concatenated hybrids. The concatenated-keys ablation at matched parameters settles it directly. If concatenated keys match physics-only keys both in-domain and under shift, the routing constraint is unnecessary and we will report that.
Why not just learn a gate that weights the physical and neural views per utterance? A gate lets the model trust one view more, and Berrebbi et al.'s finding suggests learned fusion already does this. But a gate cannot change which parts of the utterance are compared with which. Our hypothesis is about selecting comparable frames, not about view reliability. We include a gated late-fusion baseline. If it matches our model under shift, the frame-graph story is wrong.
Why not just use the frame graph with neural-feature attention, as GraphAC and ATGNN do? Those papers show frame- and patch-level graphs help on environmental audio. But their neighbor choices depend on the same representation whose reliability is in question under clinical shift. The key-source ablation compares them head to head. The prediction to confirm or refute is that the gap widens specifically on cross-condition splits.
Why not just use hand-crafted features alone, given Du et al.'s 81.8%? On small, homogeneous clinical corpora this may be nearly sufficient. It is also why we report the hand-crafted-only model. Tam et al. found deep features slightly more stable on non-speech segments, so physical features are not uniformly more robust either. The proposal succeeds only if the combination beats both single views across more than one corpus.
Proposed Experiments
In-domain evaluation covers E-DAIC (RMSE and MAE), the Mandarin dysphonia corpus (accuracy, sensitivity, specificity, subject to data access), and ARNSC classification and ALSFRS-R regression. We also run IEMOCAP with leave-one-session-out weighted F-score; the strongest number in the file is 70.30% for fine-tuned ResNet-aug x-vectors, from X-Vectors Meet Emotions.
The decisive checks are the shift conditions. On ARNSC we train on some disorder groups and recording years and test on held-out ones. On SEWA we train on German and test on Hungarian; Liu et al. report only relative gains, and no absolute CCC was found on Lacuna for this split. For emotion we train on IEMOCAP and test on CREMA-D, where x-vectors report 81.54% in-domain. We also run the controlled single-view corruption experiment described above.
The proposal is supported if physics-keyed routing beats both the neural-keyed and concatenated-keyed ablations in the shifted conditions. Its advantage should also grow with measured mismatch. We quantify mismatch as the distance between training and test WavLM embedding distributions and regress the performance gap on it. Adaptive neighborhoods should additionally beat fixed top-k while showing sizes that correlate with voicing stationarity.
The proposal is refuted if concatenated keys match physics-only keys under shift, or if gated late fusion closes the gap. In either case the useful conclusion is that the physical view's value lies in its content, not in its role as a router.
References
- Graph Attention Networks
- Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translation
- Anne Rowling Neurological Speech Corpus: clinically annotated longitudinal dataset for developing speech biomarkers in neurodegenerative disorders
- Hybrid Expert Knowledge and Self-Supervised Learning for Diagnostic Modeling of Adductor Spasmodic and Primary Myotonic Dysphonia
- Graph-Based Multi-Feature Fusion Method for Speech Emotion Recognition
- Graph Attention for Automated Audio Captioning
- ATGNN: Audio Tagging Graph Neural Network
- Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning
- ComFeAT: combination of neural and spectral features for improved depression detection
- X-Vectors Meet Emotions: A Study On Dependencies Between Emotion and Speaker Recognition