Abstract
Personalized portrait generation aims to synthesize images of a specific individual across diverse styles and environments. Current methods often struggle with identity preservation and seamless environmental blending. IC-Portrait addresses these challenges by reformulating the task as an "in-context matching" problem. By utilizing specialized modules for lighting-aware stitching and view-consistent adaptation, the framework achieves state-of-the-art results in identity consistency and relighting accuracy.
Introduction
The field of personalized portrait generation faces two primary hurdles: intra-identity diversity and the adaptation gap. User-provided reference photos often contain noise such as varied lighting, expressions, and makeup, making it difficult for models to extract a "pure" identity. Furthermore, blending a person's features into a target style—such as a neon-lit cyberpunk scene—often results in a "copy-paste" appearance or a loss of likeness.
IC-Portrait departs from traditional approaches that seek a universal mathematical representation of a face. Instead, it treats the problem as re-rendering a specific profile image into a new context, effectively bypassing the need for an abstract identity model and preserving fine-grained details.
Methodology
The IC-Portrait framework is built upon a ControlNet-like architecture, divided into two complementary processes:
1. Lighting-Aware Stitching
To bridge the adaptation gap, the model employs a high-proportion masking technique where 80% of the image is hidden during training. Inspired by Masked Autoencoders (MAE), this forces the model to learn how to "stitch" the identity into the target lighting by relying on sparse local clues and global shading. This ensures the face is not just placed in the scene but is physically influenced by the environment's illumination.
2. View-Consistent Adaptation
To handle varied poses, the model utilizes a Matching Net to establish dense pixel-to-pixel relationships between the reference profile and the target pose. This allows the model to warp reference features into the correct orientation before final generation. A key technical insight discovered by the authors is that pre-trained diffusion models are "fast learners" regarding spatial matching; while the main Appearance Net requires ~20k steps, the Matching Net achieves functional results in as few as 100–200 steps.
Datasets and Training
The researchers curated a specialized synthetic multi-view portrait dataset consisting of 35,200 view pairs. To generate high-quality ground truth data, they used celebrity names to trigger SDXL generations and employed one-shot novel-view synthesis (EG3D) to create consistent angles of the same identity. This dataset provides the necessary supervision for the model to learn how facial geometry changes across different perspectives while maintaining identity.
Key Insights
- Transformation over Generalization: By focusing on transforming a specific reference image rather than finding a mathematical "average" face, the model preserves unique skin textures and eye shapes.
- Strategic Masking: The 80% masking ratio prevents the model from simply memorizing the reference, forcing it to understand the relationship between facial features and environmental lighting.
- Spatial Alignment: Dedicated matching flows ensure that the geometry of the generated face is physically plausible, avoiding the "uncanny valley" effect common in embedding-based methods.
Results and Evaluation
IC-Portrait demonstrates significant improvements over existing baselines in both quantitative and qualitative metrics. Using the ArcFace identity similarity metric, IC-Portrait achieved a mean score of 0.565, outperforming PhotoMaker (0.351), FlashFace (0.479), and IP-Adapter-FaceID (0.480).
Qualitatively, the model excels in handling extreme lighting conditions, such as harsh shadows and colored rim lights. Additionally, because it processes faces sequentially and utilizes spatial matching, it supports multi-person scenarios without "identity bleeding," where subjects' features inadvertently mix.
Limitations and Future Work
While IC-Portrait sets a new standard for consistency, it faces certain trade-offs. Higher pixel sampling ratios can improve shading alignment but may occasionally impact generation stability. Furthermore, the model's reliance on synthetic data suggests that future iterations could benefit from more diverse real-world multi-view datasets. The authors also suggest that future research could explore more explicit semantic matching to further enhance novel-view synthesis capabilities.
Summary updated