Abstract Humans tend to form quick subjective first impressions of non-physical attributes when seeing someone's face, such as perceived trustworthiness or attractiveness. To understand what variations in a face lead to different subjective impressions, this work uses generative models to find semantically meaningful edits to a face image that change perceived attributes. Unlike prior work that relied on statistical manipulation in feature space, our end-to-end framework considers trade-offs between preserving identity and changing perceptual attributes. It maps latent space directions to changes in attribute scores, enabling a perceptually significant identity-preserving transformation of any input face along an attribute axis according to a target change. We train on real and synthetic faces, evaluate for in-domain and out-of-domain images using predictive models and human ratings, demonstrating the generalizability of our approach. Ultimately, such a framework can be used to understand and explain trends and biases in subjective interpretation of faces that are not dependent on the subject's identity. This is demonstrated with improved model performance for first impression prediction when augmenting the training data with images generated by the proposed approach for a wider range of input to learn associations between face features and subjective attributes.
Introduction
Humans form "first impressions" of others in milliseconds based solely on facial features. These judgments—perceiving someone as "trustworthy" or "dominant"—are highly subjective and often riddled with cognitive biases. In computer vision, editing objective traits like hair color or expression is well-established. However, editing subjective traits is significantly more challenging because these impressions arise from a complex interplay of features that vary across individuals.
Existing methods often struggle with a fundamental trade-off: changing a face enough to alter a subjective impression frequently results in a loss of the subject's identity. Furthermore, most datasets for first impressions contain only one image per person, making it difficult for models to distinguish between features that drive an impression and those that simply belong to a specific individual. This work addresses these challenges by proposing a framework that bridges latent image representations with human psychological ratings.
Methodology
The proposed framework utilizes a multi-stage pipeline designed to map latent face features to subjective attribute scores while maintaining identity integrity.
GAN Inversion and Latent Mapping
The system uses StyleGAN2-ADA as its generative engine. To process real-world photographs, a HyperInverter maps input images into the GAN’s latent space ($w$). The core transformation engine is a Continuous Normalizing Flow (CNF). Unlike linear transitions in latent space which often produce artifacts, the CNF learns a non-linear, continuous mapping between latent features and subjective scores. This allows for smooth, identity-aware transitions along a spectrum (e.g., a specific numerical change in trustworthiness).
Identity Preservation
To ensure the subject remains recognizable, the authors incorporate an Identity Loss mechanism. By performing PCA on the latent space, they identified Principal Editing Directions (PEDs). They discovered that freezing the first few PEDs—which control coarse features like head shape and orientation—while allowing the CNF to modify finer details effectively preserved identity while enabling subjective impression changes.
Subjective Scoring
The framework is guided by regression models (ResNet-50 and SWIN-Tiny) trained on human-annotated datasets, including the One Million Impressions (OMI) dataset for synthetic faces and the Chicago Face Dataset (CFD) for real faces. These models provide the "ground truth" scores necessary to supervise the CNF training.
Experimental Results
The framework was evaluated across multiple datasets to ensure generalizability, including SCUT-FBP5500 for attractiveness and US Adult Faces (UAF) for out-of-domain testing.
Quantitative and Human Validation
The inclusion of the "ID Loss" (freezing the first 4 PEDs) resulted in a significant improvement in Identity Similarity (IS), rising to 0.932 compared to 0.894 without the loss. Human validation further confirmed the model's efficacy; in a study with 54 participants, 67% of respondents correctly identified the "more dominant" version of a transformed face pair, aligning with the model's intended direction.
Model Boosting via Data Augmentation
One of the most significant contributions of this work is the use of synthetic transforms to improve predictive models. By generating multiple versions of the same identity with varying subjective scores, the authors created a balanced training set that forced models to ignore identity-specific features. For dominance prediction on the OMI dataset, the $R^2$ coefficient jumped from 0.163 to 0.850 when the training data was augmented with these synthetic transforms.
Discussion and Insights
The model serves as a tool for visualizing human bias. For example, the framework revealed that annotators often conflate age with dominance; increasing dominance frequently resulted in the addition of wrinkles or facial hair. Conversely, increasing trustworthiness often led to a younger appearance or a subtle smile. These findings highlight how the model reproduces social stereotypes present in the training data, such as associating specific skin tones or "baby-facedness" with certain traits.
Conclusion
This work introduces a robust framework for identity-preserving subjective face transformation. By leveraging CNFs and identity-aware latent editing, the system can successfully alter first impressions while maintaining the subject's recognizable features. Future work will focus on developing debiasing techniques to separate pure facial geometry from social stereotypes and expanding the framework to handle more complex poses and occlusions.
Summary updated