Most emotion recognition systems sort human affect into four to six broad bins such as happy, sad, angry and neutral. Newer datasets break this into 27 or 40 categories: shame versus embarrassment, contentment versus joy, a polite smile versus a genuine laugh. Low-rank adaptation (LoRA) is now the default way to fit large pretrained models to emotion tasks. LoRA freezes the pretrained weights and learns a small additive correction built from two thin matrices, so the size of the correction is capped by a chosen "rank" $r$. Every published LoRA emotion system has been tested only on coarse label sets. Nobody knows whether the small corrections that suffice for four classes still suffice for forty, or whether fine distinctions need the backbone to change in more directions.
This note proposes a controlled measurement. We train the same backbone on the same examples while varying only the granularity of the labels. The label sets are nested: 27 fine emotions, the 6 Ekman groups, and 3 sentiment groups in GoEmotions, plus analogous hierarchies in speech. For each granularity we find the smallest adapter rank at which accuracy stops improving. We also compare the learned weight changes directly. The core claim is testable. If fine-grained emotion is a real representational demand, and not just a larger output layer, then three things should hold: the saturating rank should grow with semantic granularity, the gap between LoRA and a frozen backbone with a trained output layer should widen with granularity, and the coarse-task correction should lie inside the span of the fine-task correction. If none of these hold, the field can stop tuning backbones for granularity and put its effort into heads and labels instead.
Where This Sits
The motivating evidence comes from the fine-grained emotion literature. GoEmotions built a 27-category Reddit corpus and showed with a component analysis that the categories carry dissociable signal. Its fully fine-tuned BERT-base baseline scores 0.46 macro F1 on all 27 categories, 0.64 when the categories are grouped into Ekman's six, and 0.69 at sentiment level. That drop could reflect capacity or label noise; the paper cannot tell which. EmoNet-Voice and EmoNet-Face extend fine-grained labeling to 40 categories in speech and faces, and they document an "arousal bias": experts agree far less on low-energy states such as contemplation or numbness. Any accuracy curve on these classes therefore has a human-agreement ceiling that must be separated from model capacity. Fine-Grained Facial Expression Recognition in Multiple Smiles shows that a single "happy" label hides socially distinct expressions.
The closest existing work applies LoRA to emotion but leaves label scope coarse. PEFT-SER compares LoRA, adapters, and prompt tuning on speech backbones. It restricts every dataset to four classes and reports 67.3% average UAR (unweighted average recall, the mean of per-class recalls) for WavLM Base+ with LoRA, using 0.37M trained parameters and showing little sensitivity to rank. That rank-insensitivity is exactly the observation we expect to break at finer granularity. DialogueLLM and InstructERC use LoRA on 7B-scale language models for conversational emotion recognition. They treat it as a memory saver, and InstructERC goes further by merging taxonomies back into coarse groups. DialogueLLM explicitly lists broader affect coverage as future work. Our proposal differs from all three because we treat granularity as the independent variable and the adapter's rank and subspace as the measured outcome.
The analysis method comes from Decoupling Safety into Orthogonal Subspace. That paper used singular value decomposition of LoRA updates to show that safety-refusal corrections sit largely orthogonal to the pretrained weights, while domain tasks like math did not decouple. We reuse its singular-value tooling, but we ask a different geometric question. We do not compare a correction with the pretrained weights. We compare corrections at different label granularities with each other. The tuning protocol is constrained by A Unified Study of LoRA Variants, which found that vanilla LoRA with a properly tuned learning rate matches more elaborate variants, and that LoRA's optimal learning-rate band is narrow. A rank sweep without a per-rank learning-rate sweep would therefore confound rank with mistuning.
Core Claim
Separating fine emotions requires the pretrained representation to change in more independent directions than separating coarse ones. For a backbone and dataset, define $r^*(g)$ as the smallest rank whose held-out score is within one seed-standard-deviation of the best score at granularity $g$. We predict that $r^*$ rises monotonically from sentiment to Ekman to fine labels. We predict that it does not rise for random groupings with the same number of classes. We also predict that the coarse-task update lies largely within the top singular directions of the fine-task update, with the reverse inclusion not holding.
To test inclusion, let $U_c$ and $U_f$ be the top-$k$ left singular vectors of the coarse and fine updates $\Delta W = BA$ at one layer. We measure
$$ \rho_{c\to f} = \frac{\lVert U_f^\top U_c \rVert_F^2}{k}, $$
which is the fraction of the coarse update's top directions captured by the fine update. It equals 1 for full containment and is roughly $k/d$ for unrelated random subspaces in a $d$-dimensional layer. Nesting predicts that $\rho_{c\to f}$ is high and clearly larger than $\rho_{f\to c}$.
Research Program
The experiments use two modalities where nested labels exist on identical examples. For text, we use GoEmotions with RoBERTa-base and BERT-base, at three granularities given by the paper's own Ekman and sentiment mappings. For speech, we use EmoNet-Voice Bench with WavLM Base+, where LoRA is known to behave well per PEFT-SER. We also run Whisper-Small, which PEFT-SER found misbehaves under LoRA and which EmoNet-Voice nonetheless used successfully with frozen features. Including it tests whether granularity changes the Whisper picture. For speech coarse labels we use the 40 categories grouped by valence-arousal quadrant and by an Ekman-style grouping. HUME-VB adds a third hierarchy: continuous valence and arousal, then 10 emotion dimensions. This gives a check on continuous targets.
For each backbone, granularity, and rank $r \in \{1,2,4,8,16,32,64,128\}$, we train LoRA on the attention and feed-forward projections. Each rank gets its own learning-rate sweep and three seeds. We record macro F1 (or CCC on HUME-VB, the concordance correlation coefficient that rewards matching both the trend and the scale of continuous ratings), per-class F1, and expected calibration error. We then compute singular value spectra of each layer's $\Delta W$, an effective rank given by the exponential of spectral entropy, and the inclusion scores $\rho$ between granularities.
Two controls separate semantics from class count. The first is random groupings: the 27 GoEmotions classes partitioned randomly into 6 groups and into 3 groups. These match the coarse tasks in class count but have no semantic coherence. The second is a head-capacity control. Every granularity uses the same 256-dimensional bottleneck before the output layer, so a larger output matrix cannot masquerade as backbone change. To split capacity from noise, we stratify fine classes by annotator agreement, using GoEmotions' rater correlations and EmoNet-Voice's expert consensus. Classes limited by capacity should keep improving with rank. Classes limited by label noise, such as grief and contemplation, should plateau early regardless of rank.
Baselines
The naive reference point is a frozen backbone with only the bottleneck head trained. It is tuned with the same number of learning-rate trials as any LoRA configuration. This is the strongest "no adaptation" competitor because emotion2vec reports 71.79% weighted accuracy on IEMOCAP with only a linear layer on frozen features, and EmoNet-Voice reached 70.6% on EmoDB and 74.2% on RAVDESS zero-shot with frozen Whisper-Small and per-class heads. On the frozen speech side we additionally run emotion2vec features. LoRA beats this baseline in the sense that matters only if its advantage grows with granularity. A constant gap would mean granularity is a head problem.
The strongest existing method on GoEmotions in the file is full fine-tuning of BERT-base, which reports 0.46 macro F1 on 27 classes, 0.64 on Ekman, and 0.69 on sentiment (GoEmotions). We reproduce it on the same splits with the same trial budget. The best-rank LoRA should reach parity within seed variance, and we test whether full fine-tuning's advantage, if any, is concentrated at the fine level. On HUME-VB the strongest reported system is the hierarchical regression chain at 0.7237 mean CCC (Hierarchical Regression Chain). We do not need to beat it, but a LoRA WavLM model within a few hundredths would show the measurement regime is realistic. For EmoNet-Voice Bench and EmoNet-Face in-distribution scores, no reported result was found on Lacuna, so our frozen and full fine-tuning runs define the reference.
The key ablation fixes rank at 8, the conventional setting. If rank matters at fine granularity, fixed-$r$ LoRA should fall below best-$r$ LoRA on fine labels and match it on coarse labels. A second ablation trains an automatically rank-allocating variant from the LoRA Unified Study codebase under the same budget, to check whether the rank rule we derive is already discovered implicitly.
Why not just…?
Why not just train a bigger head on frozen features? This is the most serious threat. If the frozen pretrained representation already separates shame from embarrassment, a richer head suffices, and PEFT-SER's own Whisper results show frozen features can beat LoRA. The frozen-head baseline with a matched head, together with the granularity-by-gap analysis, settles this directly. A flat gap would be a negative result we would report as the main finding.
Why not just fully fine-tune? Full fine-tuning has no rank to measure, so it cannot answer where the change lives. Prior evidence also suggests it overfits in this regime. Face forgery detection with LoRA found an 11.19-point cross-dataset AUC gain for LoRA over full ViT tuning, and On the Calibration of Large Language Models and Alignment found full fine-tuning drives calibration error above 0.20. We still run it. If it wins clearly only at fine granularity, that is itself evidence that fine emotion needs high-dimensional change.
Why not just collapse to coarse labels, where accuracy is higher? Higher accuracy on coarse bins reflects an easier task, not better understanding. The smile study and the social-assistance work of Karimi et al. show coarse bins lose socially decisive distinctions. Our random-grouping control also tests whether coarse gains are semantic or merely a function of fewer classes.
Why not just rely on rank-allocation variants to pick rank automatically? The LoRA Unified Study found such variants rarely beat tuned vanilla LoRA. More importantly, an allocator yields a number, not an explanation of whether granularity drives it. We include one as an ablation.
Decisive Checks
The proposal succeeds if three conditions hold. First, $r^*$ rises from sentiment to Ekman to fine on at least GoEmotions and EmoNet-Voice, but not for random groupings. Second, the gap between best-rank LoRA and the matched frozen head widens with granularity. Third, $\rho_{c\to f}$ substantially exceeds both $\rho_{f\to c}$ and the random-subspace level. The agreement-stratified curves then tell practitioners which fine classes deserve more rank and which are capped by human disagreement. The proposal fails informatively if $r^*$ is flat and the probe gap is constant. That outcome would say emotional nuance is already linearly present in pretrained speech and text representations, and that the next gains lie in labels and heads rather than adaptation. A secondary readout is per-class calibration on low-arousal classes as rank varies. This checks whether higher rank buys accuracy at the cost of overconfidence on exactly the states humans cannot agree on.
References
- GoEmotions: A Dataset of Fine-Grained Emotions
- EmoNet-Voice: A Large-Scale Synthetic Benchmark for Fine-Grained Speech Emotion
- EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition
- Fine-Grained Facial Expression Recognition in Multiple Smiles
- PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Models
- DialogueLLM: Context and emotion knowledge-tuned large language models for emotion recognition in conversations
- InstructERC: Reforming Emotion Recognition in Conversation with Multi-task Retrieval-Augmented Large Language Models
- Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Models
- A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
- emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
- A Hierarchical Regression Chain Framework for Affective Vocal Burst Recognition
- Enhancing General Face Forgery Detection via Vision Transformer with Low-Rank Adaptation
- On the Calibration of Large Language Models and Alignment
- FEELS: a full-spectrum enhanced emotion learning system for assisting individuals with autism spectrum disorder