GroundingFace: Fine-grained Face Understanding via Pixel Grounding Multimodal Large Language Model

Yue Han, Jiangning Zhang, Junwei Zhu, Runze Hou, Xiaozhong Ji, Chuming Lin, Xiaobin Hu, Zhucun Xue, Yong Liu

Introduction: Overcoming "Facial Blur"

Current Multimodal Large Language Models (MLLMs) excel at general scene understanding but often fail to capture the intricate details of human faces. To a standard model, a face is frequently treated as a monolithic object, missing the nuances of makeup, skin conditions, or specific accessories. This "facial blur" stems from four primary gaps identified by the researchers: insufficient attribute variety, a lack of relationship reasoning between features, sparse high-quality QA data, and low-resolution segmentation masks that typically cover only basic categories.

GroundingFace provides a "magnifying glass" for facial details, enabling precise pixel-level grounding of attributes like eyeliner and jewelry.

The GroundingFace Framework

To address these limitations, researchers developed GroundingFace, a framework designed for fine-grained facial analysis. It incorporates several architectural innovations to bridge the gap between general vision and facial precision:

  • Face Prior Sampler: This component identifies facial regions and extracts high-resolution "face tokens," allowing the model to process fine details without the prohibitive computational cost of full-image high-resolution processing.
  • HQ-SAM Adapter: By integrating a high-quality adapter into the Segment Anything Model (SAM) decoder, GroundingFace fuses shallow edge features with deep semantic features, resulting in highly accurate masks for minute elements like "crow's feet" or "nose rings."
  • LoRA Mixture of Experts (MoE): A two-stage training recipe uses MoE to route tokens to specialized experts. This ensures the model retains general scene knowledge while mastering specialized facial grounding tasks.

FacePlayGround-240K Dataset

The backbone of this research is FacePlayGround-240K, a massive dataset of 240,000 images sourced from high-quality repositories like CelebAMask-HQ and FFHQ. Unlike standard face datasets that use approximately 19 categories, FacePlayGround-240K features 47 mask categories and 7.3 million grounded regions. It employs a hierarchical structure for semantic QA, ranging from concrete attributes (e.g., hair color) to abstract traits (e.g., age) and overall impressions (e.g., mood or aesthetic style).

Key Results and Performance

GroundingFace establishes a new state-of-the-art across multiple benchmarks. In pixel-grounded captioning, it achieved a METEOR score of 23.1, a dramatic improvement over the GLaMM baseline of 1.1. Its referring segmentation capabilities are equally impressive, with a gIoU of 88.9, showing a massive leap in finding specific features mentioned in text.

Performance comparison of GroundingFace against leading MLLMs across various facial understanding metrics.

As shown in the radar chart above, GroundingFace consistently outperforms models like Qwen-VL and LLaVA-v1.5 in emotion accuracy, attribute recognition, and age estimation, even in zero-shot scenarios. It demonstrates a unique ability to perform reasoning-based grounding—explaining why a person looks a certain way by pointing to specific combinations of facial features.

Conclusion

GroundingFace represents a significant step forward in facial AI, moving beyond simple identification to deep, grounded reasoning. By combining high-resolution sampling with a massive, granular dataset, it enables models to "see" and explain the subtle details that define human appearance and expression. While computational costs and data imbalances remain challenges, the framework opens new doors for video-based face understanding and complex social-relation reasoning.

Summary updated

Abstract

Multimodal Language Learning Models (MLLMs) have shown remarkable performance in image understanding, generation, and editing, with recent advancements achieving pixel-level grounding with reasoning. However, these models for common objects struggle with fine-grained face understanding. In this work, we introduce the FacePlayGround-240K dataset, the first pioneering large-scale, pixel-grounded face caption and question-answer (QA) dataset, meticulously curated for alignment pretraining and instruction-tuning. We present the GroundingFace framework, specifically designed to enhance fine-grained face understanding. This framework significantly augments the capabilities of existing grounding models in face part segmentation, face attribute comprehension, while preserving general scene understanding. Comprehensive experiments validate that our approach surpasses current state-of-the-art models in pixel-grounded face captioning/QA and various downstream tasks, including face captioning, referring segmentation, and zero-shot face attribute recognition.
BibTeX
@inproceedings{han2025groundingface,
  author = {Yue Han and Jiangning Zhang and Junwei Zhu and Runze Hou and Xiaozhong Ji and Chuming Lin and Xiaobin Hu and Zhucun Xue and Yong Liu},
  title = {{GroundingFace: Fine-grained Face Understanding via Pixel Grounding Multimodal Large Language Model}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00373},
}
Figure