Contour Shape as a Free Teacher for Speech Emotion Recognition

Emotion in speech depends heavily on how pitch and loudness move over time: whether a syllable rises or falls, whether it peaks or dips. Large pretrained speech encoders such as wav2vec 2.0 are now the default starting point for recognizing emotion, but they are fine-tuned only on emotion labels, sometimes with transcripts added as a second objective. This proposal adds a different second objective. While learning emotion, the network must also predict the shape of the pitch and energy curve inside each short syllable-like chunk of audio. Each shape is summarized by a handful of numbers: overall level, slope, curvature, and a few finer wiggles. These numbers come from fitting orthogonal polynomials, which are curves built so that each coefficient measures one geometric property independently of the others. The targets are computed automatically from the waveform, so no transcripts or human annotations are required.

The intended contribution is not simply "add a pitch loss." Predicting the pitch curve frame by frame is mathematically equivalent to predicting all polynomial coefficients with equal weight, including the level term and the high-order terms that mostly reflect pitch-tracker noise. Moving to the coefficient domain lets the auxiliary task keep only the shape properties that carry emotional information and weight them deliberately. The proposal tests whether shape-selective supervision helps a multi-task emotion model where plain pitch regression, summary statistics, and transcript-based auxiliaries do not. It also asks which shape orders account for the gain.

Core Claim

Let the log-$F_0$ (pitch) contour of a segment be $f(t)$, sampled at the segment's voiced frames. Map those frames onto the interval $[-1,1]$ and expand the contour in a discrete orthonormal Legendre basis $\{P_k\}$. This basis is built by Gram–Schmidt orthogonalization of $1, t, t^2, \dots$ on the actual sample points, so orthogonality holds exactly even when frames are missing. The expansion gives coefficients $c_k = \sum_t f(t) P_k(t)$. By orthonormality, the frame-level squared error between a predicted contour $\hat f$ and the true contour decomposes as

$$\sum_t \big(f(t)-\hat f(t)\big)^2 = \sum_{k=0}^{K} (c_k-\hat c_k)^2 + r,$$

where $K$ is the truncation order and $r$ is the residual energy above order $K$. Frame-level pitch regression therefore uses fixed, equal weights over every order, and the $c_0$ level term dominates that sum. Level is largely speaker identity, and the residual is largely tracker noise. Our auxiliary loss replaces the fixed weights with chosen ones:

$$L_{\text{total}} = L_{\text{SER}} + \sum_{k=1}^{K} \lambda_k \,\rho(c_k-\hat c_k),$$

Here $L_{\text{SER}}$ is the emotion cross-entropy and $\lambda_k$ is the weight on order $k$. $c_0$ is either speaker-normalized or dropped. $\rho$ is an absolute-error penalty, following the evidence that absolute-error contour fitting resists pitch halving and doubling better than squared error.

The claim is falsifiable. If the emotion gains come from shape specifically, they should concentrate in the slope and curvature orders. Equal-weight frame regression should do no better than having no auxiliary task. Because the coefficients are orthogonal, each order can be ablated independently. That is the property the polynomial basis contributes that a learned latent cannot.

Grounding in Prior Work

The method comes from the classical polynomial-contour literature. Dehak et al. segment speech into pseudo-syllables at energy valleys (at least 60 ms) without transcripts and fit 6th-order Legendre polynomials to log pitch, log energy, and formants. Dumouchel et al. fit up to 5th-order Legendre curves to pitch and energy for child emotion recognition, reaching 70.84% weighted recall with the contour system alone on 2-class FAU AIBO. We reuse their segmentation and fitting. The difference is in how the coefficients are used. Both papers feed them as input features to separate classifiers and fuse decisions afterwards. We use them as training targets that shape a shared neural encoder, which is then deployed without any contour pipeline at test time.

Yarra and Ghosh show that minimum-absolute-error polynomial stylization recovers up to 6.49 Hz of frame RMSE over squared-error fitting under noise. This motivates using absolute-error fitting both when computing targets and as the training penalty. Bone et al. find clinically relevant differences in syllable-level slope and curvature that scalar averages miss. That supports our hypothesis that the informative orders are 1 and 2.

The closest neural use of Legendre coefficients is Corkey et al.. They add a coefficient predictor for $c_0, c_1, c_2$ to FastPitch for controllable synthesis, and conditioning on ground-truth coefficients lowers $F_0$ RMSE from 55.143 Hz to 47.370 Hz on LJSpeech. That work is single-task and single-speaker, and its coefficients condition a generator. Ours are predicted from audio to regularize a recognizer, across many speakers.

On the multi-task side, two systems set the bar. Speech Emotion Recognition with Multi-Task Learning jointly trains wav2vec 2.0 on emotion and on transcription with a connectionist temporal classification loss, an alignment-free transcription objective, and reports 78.15% weighted accuracy (WA) on 4-class IEMOCAP. Hsu et al. decompose speech into content, timbre, pitch, and rhythm latents with autoencoders and train on four tasks, reaching 77.50% unweighted accuracy (UA). Both show that auxiliary objectives can help emotion recognition. Both depend on either transcripts or unconstrained latents whose individual dimensions cannot be interpreted or ablated. We replace these with a transcript-free target whose components are independent by construction.

Two lines of work constrain the design. The redundancy finding in Dehak et al. is that formant trajectories added little once cepstral features were present. This warns that contour information may already be present in the encoder, so we test for it directly with probes rather than assume otherwise. The hierarchical prosody models of Sun et al. and Chien & Lee indicate that prosody spans multiple timescales. We keep a single pseudo-syllable scale in the main study and treat the utterance scale only as an ablation, so that one variable changes at a time.

Research Program

Target extraction is offline. We run a pitch tracker and frame energy on each utterance, segment it into energy-valley pseudo-syllables, and fit orders 0 through 5 of discrete orthonormal Legendre polynomials to voiced log-$F_0$ and to log energy using absolute-error fitting. For each segment we normalize $c_0$ by the speaker's mean, using training-set speakers only.

The model is a wav2vec 2.0 encoder with two kinds of heads. One is an utterance-level emotion head. The other is a small contour head that mean-pools encoder frames over each pseudo-syllable span and regresses that segment's coefficient vector. The contour head is discarded at test time, so inference cost is unchanged.

We first fix $\lambda_k$ by a small, budget-matched search over three settings: shape-only (orders 1–2), all orders equal, and all orders except 0. Because the order losses are orthogonal, each order can also be treated as its own auxiliary task. A second variant therefore learns the per-order weights with the convex task-weighting idea from Situ et al., which drives unhelpful auxiliaries toward zero weight. The learned weights then serve as a measurement of which shape orders matter.

Finally, we analyze interference. We log the cosine similarity between each order's auxiliary gradient and the emotion gradient on the shared encoder. Gradient-rotation methods such as RotoGrad are held in reserve and used only if conflict is observed, because of their compute overhead.

Baselines

The naive reference point is wav2vec 2.0 fine-tuned on emotion labels alone, using the same encoder checkpoint, optimizer, epochs, and leave-one-session-out IEMOCAP protocol as every other system. Every auxiliary method has to beat it to justify any added complexity.

The strongest existing competitor is the joint transcription-plus-emotion model of Speech Emotion Recognition with Multi-Task Learning, which reports 78.15% WA on 4-class IEMOCAP. The decomposed-speech model of Hsu et al. reports 77.50% UA on the same 4-class setup. IEMOCAP splits and class mappings vary across papers. We will therefore reimplement the transcription auxiliary in our pipeline and report both our reproduction and the published number, and we will judge our method against the reproduction. We count it as beating this baseline if the contour auxiliary alone, which needs no transcripts, matches the reproduced transcription-auxiliary model within seed variance, or if contour plus transcription exceeds transcription alone by more than the standard deviation across five seeds.

The key ablation is frame-level pitch-and-energy regression as the auxiliary task. By the decomposition above, this baseline is our loss with equal weights on all orders plus the residual. It isolates the claim that coefficient selection, not contour supervision as such, provides the benefit. It receives the same tuning budget for its single loss weight. Two further baselines complete the set. The first is an auxiliary that predicts per-segment scalar statistics (mean, range, standard deviation), following the FastPitch-style averaged pitch of Łańcucki. The second is input fusion, in which the same Legendre coefficients are concatenated to pooled encoder features rather than predicted, in the spirit of Dumouchel et al..

For the child-speech setting on 2-class FAU AIBO, the reference is the fused cepstral-plus-Legendre system of Dumouchel et al., at 70.29% unweighted average recall (UAR). We report it as a historical anchor, and our own single-task wav2vec baseline serves as the fair comparison.

Why not just…?

Why not just regress the frame-level pitch curve as the auxiliary task? This is the most serious threat, because it is simpler and contains the same information. Our argument is the decomposition: frame regression gives fixed weight to speaker-dependent level and to tracker noise, and it uses squared error, which Yarra and Ghosh show is distorted by octave errors. If frame regression matches shape-only coefficient supervision on clean IEMOCAP, the selection idea is unnecessary there and we will report that. The noise-injection experiment below is designed to separate the two.

Why not just feed the Legendre coefficients in as extra input features? Input fusion is what the classical literature did. Dehak et al. found such added contour features largely redundant with spectral ones. Input fusion also needs a pitch tracker at test time and inherits its failures. An auxiliary target changes the encoder during training and is dropped at inference. The input-fusion baseline, run with identical data and tuning, tests which approach transfers better, especially under noise, where test-time tracking fails.

Why not just use the transcription auxiliary, which already works? Speech Emotion Recognition with Multi-Task Learning shows it helps, but it requires transcripts, which emotional and child corpora often lack. It also pushes the encoder toward linguistic content rather than prosody. We expect the two auxiliaries to be complementary, and the contour-plus-transcription condition tests this. If transcription alone dominates and the contour target adds nothing, the transcript-free advantage remains, but only for unlabeled corpora.

Why not trust that wav2vec 2.0 already encodes contour shape? It may, which is the modern form of the redundancy finding. Before training, we fit linear probes from frozen encoder layers to each coefficient order. If slope and curvature are already linearly decodable with high R², we predict a small benefit from the auxiliary. The probe results thus give a prediction that the multi-task results can confirm or refute.

Proposed Experiments

The main experiment is 4-class IEMOCAP with leave-one-session-out evaluation, five seeds, and both WA and UA, comparing all baselines above. FAU AIBO 2-class UAR tests transfer to child speech, where transcripts are scarce.

The first decisive check is the per-order ablation. We train with only order 0, only order 1, only order 2, and orders 3–5. The hypothesis predicts gains for orders 1 and 2 only, and the learned task weights from the Situ-style variant should agree with that ranking.

The second check is robustness. We add babble and white noise at several signal-to-noise ratios to the training audio that targets are computed from, and compare absolute-error versus squared-error target fitting, and coefficient versus frame-level auxiliaries. We expect the coefficient auxiliary with absolute-error fitting to degrade least.

The third check is speaker leakage. We train a speaker-identification probe on the fine-tuned encoder and test whether including $c_0$ increases speaker information while decreasing emotion UA. If so, that supports dropping or normalizing $c_0$.

A positive outcome looks like this: shape-only supervision beats both the frame-regression and statistics auxiliaries across seeds, the gain concentrates in orders 1–2, and the margin over frame regression grows under noise. A negative outcome, where all contour auxiliaries tie, would still be informative. It would indicate that pretrained speech encoders already make explicit contour supervision unnecessary for emotion recognition.