Transformation of Emotions in Images Using Poisson Blended Generative Adversarial Networks (Student Abstract)

Aristidis Dernelakis, Jungin Kim, Kevin Velasquez, Lee StearnsAAAI 2022 · DOI

Introduction

Generative Adversarial Networks (GANs) have demonstrated remarkable success in specialized image manipulation tasks, such as altering facial expressions or applying artistic styles. However, these models often struggle with "unconstrained" real-world images containing complex backgrounds, multiple subjects, or varied lighting. While global style transfer models often neglect fine-grained facial details, face-specific GANs can produce "uncanny" results where the modified face fails to blend naturally with the surrounding environment. This research proposes a multi-stage pipeline that bridges this gap by combining local facial manipulation with global stylistic adjustments.

Methodology: A Modular Pipeline

Rather than relying on a single end-to-end model, the authors developed a three-stage process to ensure both detail and cohesion:

  1. Facial Transformation: The system utilizes StarGAN-v2 to modify the facial expressions of subjects within an image to a target emotion (e.g., from anger to fear).
  2. Poisson Blending: To integrate the modified faces back into the original scene without harsh edges or lighting inconsistencies, Poisson blending is applied. This ensures the gradients of the new facial features match the original image context.
  3. Global Style Transfer: Finally, a fast style-transfer network adjusts the broader visual elements—such as color saturation and contrast—to align with a reference image representing the target emotion.
The proposed multi-stage pipeline combining StarGAN, Poisson Blending, and Style Transfer.

Data Curation and Quality

The researchers utilized the WEBEmo dataset, comprising over 267,000 stock photos. Recognizing that image-level labels are often noisy for fine-grained facial training, an automated curation pipeline was implemented using YOLO v3 for person detection and FaceNet for precise facial cropping. An FER (Facial Expression Recognition) classifier was then used to filter crops based on confidence scores, ensuring the GAN learned from high-quality, representative examples.

Results and Discussion

The proposed method demonstrates that emotion is holistic, captured through both facial micro-expressions and global stylistic cues.

StarGAN-v2 results showing consistent identity preservation across multiple target emotions.

Qualitative results indicate that the "divide and conquer" strategy effectively leverages the strengths of specialized architectures. As shown in the pipeline overview, transforming an "angry" scene into a "fearful" one involves not just facial adjustments (like furrowed brows) but also a shift toward darker, desaturated tones that reinforce the emotional narrative. This multi-stage approach creates a more cohesive emotional "story" than a face-change alone.

Conclusion and Future Work

This modular approach provides a robust framework for emotional transformation in complex images. Future research will focus on automating the selection of style reference images and expanding the pipeline to handle more complex scene elements, such as body pose and object-level modifications, to further enhance emotional realism.

Summary updated

BibTeX
@inproceedings{dernelakis2022transformation,
  author = {Aristidis Dernelakis and Jungin Kim and Kevin Velasquez and Lee Stearns},
  title = {{Transformation of Emotions in Images Using Poisson Blended Generative Adversarial Networks (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21603},
}
Figure