Abstract
Humans can communicate emotions through a plethora of facial expressions, each with its own intensity, nuances and ambiguities. The generation of such variety by means of conditional GANs is limited to the expressions encoded in the used label system. These limitations are caused either due to burdensome labeling demand or the confounded label space. On the other hand, learning from inexpensive and intuitive basic categorical emotion labels leads to limited emotion variability. In this paper, we propose a novel GAN-based framework which learns an expressive and interpretable conditional space (usable as a label space) of emotions, instead of conditioning on handcrafted labels. Our framework only uses the categorical labels of basic emotions to jointly learn the conditional space as well as the emotion manipulation. Such learning can benefit from the image variability within discrete labels, especially when the intrinsic labels reside beyond the discrete space of the defined. Our experiments demonstrate the effectiveness of the proposed framework, by allowing us to control and generate a gamut of complex and compound emotions, while using only the basic categorical emotion labels during training.
Introduction
Most conditional GANs for facial expression synthesis rely on rigid, discrete labels. If a user wants a "happy" face, they provide a one-hot vector. However, human emotion is a spectrum; a person isn't just "angry," they might be "slightly annoyed" or "furious." Furthermore, emotions often blend into compound expressions. Labeling spontaneous, real-world emotions with precise intensity or blend scores is impractical and subjective. GANmut addresses this by learning a rich, continuous, and interpretable emotional space using only easily obtainable categorical labels.
Methodology: Learning the Emotional Space
The GANmut framework replaces standard fixed label inputs with a learnable 2D conditional space ($Z$). The architecture, built upon StarGAN, consists of a Generator ($G$) and a Discriminator ($D$).
The framework employs two primary models for structuring this space:
- The Linear Model: Uses polar coordinates $(\theta, \rho)$, where the angle ($\theta$) represents the emotion category and the radius ($\rho$) represents the intensity.
- The Gaussian Model (GGANmut): Models the space as a Mixture of Gaussians, where each basic emotion is a "mode." This is particularly effective for representing compound emotions by sampling from regions between Gaussian peaks.
The training logic relies on three key losses: a Classification Loss to ensure categorical accuracy, a Regression Loss where the discriminator predicts the exact 2D coordinates ($z$) to ensure consistency, and a Smoothness Constraint to encourage clearer expressions as intensity increases.
Visualizing the Emotional Gamut
The resulting learned space is highly interpretable. By moving along different axes or radii, the model can generate a continuous range of expressions.
As shown in the figure, the model naturally learns to correlate distance from the origin with visual intensity (e.g., a wider smile). Interestingly, the "empty" space between vectors like "Angry" and "Disgusted" naturally learns to represent blends, allowing the model to discover compound emotions without explicit supervision.
Key Results and Insights
The models were evaluated on AffectNet, containing roughly 450,000 spontaneous expressions.
- Frechet Emotion Distance (FED): GANmut achieved a FED of 0.71, significantly outperforming StarGAN (4.89) and SMIT (1.89), indicating that its generated distribution closely matches real emotions.
- Smoothness: Quantitative analysis showed that GANmut produces much smoother transitions in intensity compared to baselines like StarGAN, which often "flip" abruptly between states.
- Emergent Interpretability: The learned 2D space often resembles the psychological "Valence-Arousal" model, where similar emotions cluster together naturally.
Limitations and Future Work
The learned latent space is influenced by data distribution; for instance, the "Happy" region in AffectNet is disproportionately large. Additionally, while 2D is interpretable, human emotion may be higher-dimensional. Future work may also address identity preservation at extreme emotional intensities, which remains a challenge for many GAN-based editors.
Summary updated