IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait

Han Yang, Enis Simsar, Sotiris Anagnostidis, Yanlong Zang, Thomas Hofmann, Ziwei LiuarXiv 2025 · DOI · arXiv

Abstract

Personalized portrait generation aims to synthesize images of a specific individual across diverse styles and environments. Current methods often struggle with identity preservation and seamless environmental blending. IC-Portrait addresses these challenges by reformulating the task as an "in-context matching" problem. By utilizing specialized modules for lighting-aware stitching and view-consistent adaptation, the framework achieves state-of-the-art results in identity consistency and relighting accuracy.

Introduction

The field of personalized portrait generation faces two primary hurdles: intra-identity diversity and the adaptation gap. User-provided reference photos often contain noise such as varied lighting, expressions, and makeup, making it difficult for models to extract a "pure" identity. Furthermore, blending a person's features into a target style—such as a neon-lit cyberpunk scene—often results in a "copy-paste" appearance or a loss of likeness.

IC-Portrait departs from traditional approaches that seek a universal mathematical representation of a face. Instead, it treats the problem as re-rendering a specific profile image into a new context, effectively bypassing the need for an abstract identity model and preserving fine-grained details.

Methodology

The IC-Portrait framework is built upon a ControlNet-like architecture, divided into two complementary processes:

1. Lighting-Aware Stitching

To bridge the adaptation gap, the model employs a high-proportion masking technique where 80% of the image is hidden during training. Inspired by Masked Autoencoders (MAE), this forces the model to learn how to "stitch" the identity into the target lighting by relying on sparse local clues and global shading. This ensures the face is not just placed in the scene but is physically influenced by the environment's illumination.

2. View-Consistent Adaptation

To handle varied poses, the model utilizes a Matching Net to establish dense pixel-to-pixel relationships between the reference profile and the target pose. This allows the model to warp reference features into the correct orientation before final generation. A key technical insight discovered by the authors is that pre-trained diffusion models are "fast learners" regarding spatial matching; while the main Appearance Net requires ~20k steps, the Matching Net achieves functional results in as few as 100–200 steps.

Demonstration of IC-Portrait's consistency across varying poses (top) and diverse lighting/environmental contexts (bottom).

Datasets and Training

The researchers curated a specialized synthetic multi-view portrait dataset consisting of 35,200 view pairs. To generate high-quality ground truth data, they used celebrity names to trigger SDXL generations and employed one-shot novel-view synthesis (EG3D) to create consistent angles of the same identity. This dataset provides the necessary supervision for the model to learn how facial geometry changes across different perspectives while maintaining identity.

Key Insights

  • Transformation over Generalization: By focusing on transforming a specific reference image rather than finding a mathematical "average" face, the model preserves unique skin textures and eye shapes.
  • Strategic Masking: The 80% masking ratio prevents the model from simply memorizing the reference, forcing it to understand the relationship between facial features and environmental lighting.
  • Spatial Alignment: Dedicated matching flows ensure that the geometry of the generated face is physically plausible, avoiding the "uncanny valley" effect common in embedding-based methods.

Results and Evaluation

IC-Portrait demonstrates significant improvements over existing baselines in both quantitative and qualitative metrics. Using the ArcFace identity similarity metric, IC-Portrait achieved a mean score of 0.565, outperforming PhotoMaker (0.351), FlashFace (0.479), and IP-Adapter-FaceID (0.480).

Personalized generation across distinct environments, showcasing identity preservation and seamless blending.

Qualitatively, the model excels in handling extreme lighting conditions, such as harsh shadows and colored rim lights. Additionally, because it processes faces sequentially and utilizes spatial matching, it supports multi-person scenarios without "identity bleeding," where subjects' features inadvertently mix.

Limitations and Future Work

While IC-Portrait sets a new standard for consistency, it faces certain trade-offs. Higher pixel sampling ratios can improve shading alignment but may occasionally impact generation stability. Furthermore, the model's reliance on synthetic data suggests that future iterations could benefit from more diverse real-world multi-view datasets. The authors also suggest that future research could explore more explicit semantic matching to further enhance novel-view synthesis capabilities.

Summary updated

Abstract

Existing diffusion models show great potential for identity-preserving generation. However, personalized portrait generation remains challenging due to the diversity in user profiles, including variations in appearance and lighting conditions. To address these challenges, we propose IC-Portrait, a novel framework designed to accurately encode individual identities for personalized portrait generation. Our key insight is that pre-trained diffusion models are fast learners (e.g.,100 ~ 200 steps) for in-context dense correspondence matching, which motivates the two major designs of our IC-Portrait framework. Specifically, we reformulate portrait generation into two sub-tasks: 1) Lighting-Aware Stitching: we find that masking a high proportion of the input image, e.g., 80%, yields a highly effective self-supervisory representation learning of reference image lighting. 2) View-Consistent Adaptation: we leverage a synthetic view-consistent profile dataset to learn the in-context correspondence. The reference profile can then be warped into arbitrary poses for strong spatial-aligned view conditioning. Coupling these two designs by simply concatenating latents to form ControlNet-like supervision and modeling, enables us to significantly enhance the identity preservation fidelity and stability. Extensive evaluations demonstrate that IC-Portrait consistently outperforms existing state-of-the-art methods both quantitatively and qualitatively, with particularly notable improvements in visual qualities. Furthermore, IC-Portrait even demonstrates 3D-aware relighting capabilities.
BibTeX
@misc{yang2025ic,
  author = {Han Yang and Enis Simsar and Sotiris Anagnostidis and Yanlong Zang and Thomas Hofmann and Ziwei Liu},
  title = {{IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait}},
  year = {2025},
  eprint = {2501.17159},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.17159},
}
Figure