Who Pays for a Shared Face Backbone? Class-Conditional Conflict in Joint Identity and Expression Models

Face analysis systems are increasingly built as one network that does many jobs at once: recognize who a person is, estimate age, predict attributes, and read facial expression. Such systems share a single feature extractor (the "backbone") across tasks. It is well documented that the identity task and the expression task pull that backbone in opposite directions. Identity wants features that ignore a person's momentary expression, while expression wants exactly those momentary muscle movements. What nobody has measured is which expressions absorb the damage. Expression datasets are heavily skewed: happy and neutral faces are common, and fear and disgust are rare. Our hypothesis is that the cost of sharing is not spread evenly. Rare emotions contribute only a sliver of the expression gradient, so the identity objective overwrites the features they depend on first. If that is true, then headline accuracy for multi-task face models hides a minority-class collapse.

The proposal has two parts. First, we measure negative transfer separately for each emotion class in a joint identity-and-expression model. Negative transfer here means how much worse each class becomes when the identity task shares the backbone. Second, if the damage concentrates on rare classes, we test a targeted fix: class-conditional gradient projection. During training, we remove the part of the identity gradient that directly opposes the gradient of each rare emotion class on the shared layers. The decisive question is whether this beats the obvious remedies, namely reweighting the expression loss, freezing a fixed classifier geometry, or splitting the networks entirely, on rare-class recall at equal cost.

Core Claim

The detection-identification literature treats conflict between tasks as a property of the whole task. RelationTrack splits shared tracking features into a detection stream and an identity stream because a single representation "underperforms on both sub-tasks". Towards Fully Decoupled End-to-End Person Search goes further and trains separate side-networks, freezing the detector before training identification. In faces, SwinFace frames face recognition versus expression recognition as identity-invariant versus muscle-movement features and resolves it with per-task channel attention. FaceXFormer reports that its unified model underperforms single-task face recognition because of this same conflict.

All four papers motivate the proposal, and all four resolve conflict between tasks, never within a task. Meanwhile, the class-imbalance literature shows rare emotions are the first representations a network erases. Le Moine et al. call this class collapse. Islam et al. show performance falls apart on surprise and disgust while staying high on joy. Creanga et al. show transformers overpredict neutral.

Our claim joins these: inside a shared backbone, the competing identity objective acts as an extra majority voice against rare emotion classes. Write $g_{\text{id}}$ for the identity-loss gradient on the shared parameters and $g_c$ for the expression-loss gradient computed only on samples of emotion $c$. The prediction is that the conflict $\cos(g_{\text{id}}, g_c)$ is more negative, and more persistent through training, for rare classes than for common ones. The prediction also says per-class negative transfer is largest for fear and disgust. This is falsifiable. If negative transfer turns out uniform across classes, then the problem is ordinary task conflict plus ordinary imbalance, and existing fixes suffice.

One apparent contradiction must be faced directly. SwinFace reaches 90.97% accuracy on RAF-DB despite sharing with face recognition, which suggests sharing costs expression little. But overall accuracy on RAF-DB is dominated by happy and neutral faces, and the file contains no per-class or macro-averaged numbers for SwinFace. The reported figure is therefore compatible with our hypothesis rather than evidence against it.

Research Program

The base system is a reimplementation of the SwinFace design, restricted to two heads so the effect is legible. The first is a Swin-T backbone with a face-recognition head trained with a CosFace margin loss on MS1MV3 identities. The second is an expression head trained on RAF-DB's seven basic classes, and separately on AffectNet. Both heads use the multi-level channel attention module SwinFace introduced.

The diagnostic stage trains three matched variants: expression-only, identity-only, and joint. It then records three things:

  1. Per-class recall gaps between the expression-only and joint models.
  2. Per-class gradient cosines $\cos(g_{\text{id}}, g_c)$ at each of the four backbone stages throughout training.
  3. Linear-probe recall for each emotion on frozen features from each stage.

The probes tell us whether rare-class information is actually missing from the shared features, or merely ignored by the head. Recording gradient cosines per stage also tests SwinFace's observation that subtle attributes prefer intermediate feature maps. The question is whether rare emotions are erased specifically at the deepest stage.

The intervention stage adds class-conditional projection. At each step, for every emotion class $c$ whose gradient conflicts with identity, we remove that conflicting component:

$$g_{\text{id}} \leftarrow g_{\text{id}} - \lambda_c \,\frac{\min(0,\, g_{\text{id}}\cdot g_c)}{\|g_c\|^2}\, g_c$$

Here $\lambda_c \in [0,1]$ grows with the inverse training frequency of class $c$, so the protection is strongest for the rarest classes. The identity gradient is changed only on shared parameters; the identity head is untouched. The cost is one extra backward pass per class present in the batch. We confine the projection to the last two backbone stages, which bounds the overhead with seven classes; we will report wall-clock time per epoch.

To cover the "detection" half of the trade-off, a secondary study tests the paradox flagged in the file's gap analysis. Trident Person Search scales identification gradients by detection confidence, an idea it calls the Adaptive Gradient Weighting Function. We feed expression crops from a face detector and apply the same confidence-scaled gradients to the expression head. We then check whether rare and subtle expressions receive systematically lower detector confidence, and therefore smaller gradients. This determines whether confidence gating, a standard noise remedy in joint detection-identification pipelines, silently worsens class collapse.

Baselines

The naive reference point is an expression-only Swin-T trained with plain cross-entropy on the same data. It shows what rare-class recall looks like with no identity task and no imbalance remedy.

The strongest existing multi-task competitor is the SwinFace design trained jointly with cross-entropy. SwinFace reports 90.97% accuracy on RAF-DB, but no reported result was found on Lacuna for its macro-averaged or per-class recall. We therefore reimplement it with the same backbone, data, schedule, and seed set, and report both metrics. Beating it means higher macro-averaged recall and higher fear and disgust recall, with overall accuracy within one point and LFW verification within 0.2 points.

The strongest imbalance-specific competitor is the fixed-weight ArcFace head of Xu et al.. It freezes uniformly spread class centers on a hypersphere so majority gradients cannot warp them. It reports 84.485% RAF-DB accuracy with ResNet18 and 70.828% on the FER2013 private test, along with recall gains on disgust and fear. We run it both single-task and as the expression head inside the joint model, which tests whether frozen class geometry survives a competing identity loss deforming the features beneath it. We also run joint training with inverse-frequency class weighting, following the approach Bourbour et al. found effective.

The architectural competitor is full decoupling in the style of Zhang et al.: a frozen identity network plus a separate expression side-network. We match its total parameter count against a widened joint model.

The key ablation is class-agnostic projection. It removes the conflict between the identity gradient and the total expression gradient, the standard whole-task gradient-surgery idea. Any gain of our method over this ablation is attributable to conditioning on class rather than to gradient surgery in general.

All methods get the same number of hyperparameter trials, on the same validation split, at matched compute. Results are averaged over five seeds with confidence intervals.

Why not just…?

Why not just reweight the expression loss by class frequency? This is the most serious threat. Reweighting enlarges $g_c$ for rare classes, which could outvote the identity gradient without any projection. Two findings make us doubt it is enough. Bogatyreva et al. find that weighted losses combined with augmentation still give poor recall on underrepresented emotions. Reweighting also scales the rare-class gradient without changing its direction relative to $g_{\text{id}}$, so the opposing component remains. The direct test is the joint model with reweighting versus the same model with projection. If reweighting closes the per-class negative-transfer gap, we will report that and the method is unnecessary.

Why not just use separate networks, since sharing is the cause? Full decoupling removes the conflict by construction, and if it matches our method at equal parameters it is the simpler answer. Its cost is the parameter and memory overhead that Zhang et al. themselves incur. It also forgoes whatever identity pretraining contributes to scarce tasks, an effect SwinFace observed for age estimation. The parameter-matched comparison settles this, and we commit to reporting it either way.

Why not just freeze the classifier geometry with fixed-weight ArcFace? Fixed centers stop majority classes from crowding the classifier, but they constrain only the last layer. The identity loss acts on the backbone below. Running Xu et al.'s head inside the joint model directly shows whether rare-class recall still drops when the identity task is added.

Why not just remove identity from expression features, as ELIM or CAFE do? ELIM subtracts an identity shift found by optimal transport, a method for matching samples across identities at minimum cost. CAFE masks frozen CLIP channels. Neither trains an identity head at the same time, so neither addresses a deployed multi-task model. Both also leave imbalance open: ELIM was evaluated only on continuous valence and arousal, and CAFE stays weak on AffectNet, which its authors blame on noise and imbalance. We include CAFE's RAF-DB-to-FERPlus transfer setting, where it reports 73.16%, as a cross-dataset check rather than a head-to-head competitor.

Proposed Experiments

The first decisive check is the diagnostic itself. On RAF-DB and AffectNet, we test whether the per-class recall drop from expression-only to joint training correlates negatively with class frequency, and whether the rare-class gradient cosines stay negative after the common classes' cosines have recovered. A flat profile falsifies the core claim and ends the intervention study.

The second check compares projection against reweighting, fixed-weight ArcFace, decoupling, and class-agnostic projection. The comparison uses macro-averaged recall and fear and disgust recall on RAF-DB, AffectNet, and FER2013, while tracking LFW verification to make sure identity is not quietly sacrificed.

The third check tests robustness across datasets. We train on RAF-DB and evaluate on FERPlus and SFEW 2.0 to see whether protected rare-class features transfer or merely fit one dataset's labeling conventions. The latter is a risk raised by A Deeper Look at Facial Expression Dataset Bias, which shows annotators disagree on what counts as disgust versus anger.

The fourth check is the confidence-gating study. We correlate detector confidence with emotion class and measure rare-class recall with and without confidence-scaled gradients.