TL;DR
Hanfu-Bench is a new expert-curated multimodal dataset designed to evaluate how well vision-language models (VLMs) understand cultural evolution over time rather than just across geographic regions. By testing models on traditional Chinese Hanfu clothing, the benchmark reveals that even advanced VLMs struggle significantly with multi-image historical reasoning and fail to effectively blend traditional cultural elements into modern fashion designs.
What Problem Does This Paper Address?
While recent AI research has focused on making vision-language models culturally aware, these efforts primarily look at geographic or spatial diversity—such as recognizing food or traditional clothing from different countries. However, culture also evolves temporally within the same region.
To capture this overlooked dimension, the authors focus on Hanfu, the traditional attire of the Han Chinese, which features distinct structural variations (sleeve styles, collar designs, and garment layers) across different historical dynasties. Understanding Hanfu requires fine-grained historical and aesthetic knowledge. Furthermore, modern Chinese society frequently adapts Hanfu into contemporary fashion, making it an ideal testbed for both cultural visual understanding and creative adaptation (or "transcreation").
Methods
The authors established a rigorous data collection and evaluation pipeline consisting of two primary tasks:
- Cultural Visual Understanding: This task uses automatically generated multiple-choice questions based on expert annotations. It is split into Single-Image Visual Question Answering (SVQA) and Multi-Image VQA (MVQA) across eight distinct feature categories (e.g., dynasty period, collar type, sleeve style).
- Cultural Image Transcreation: This task evaluates a model's ability to transform traditional Hanfu into modern, wearable clothing (like hoodies or trench coats) while preserving its cultural essence. The authors implement a cascaded framework: GPT-4o generates a detailed caption of a traditional Hanfu image, edits the caption to incorporate modern design requirements, and then passes the edited text to an image generation model.
Datasets
The paper introduces Hanfu-Bench, which contains 496 distinct Hanfu outfits and 1,192 high-quality images.
- Sources: Sourced online via web scraping from popular Taobao retailers and manual screenshots from critically acclaimed Chinese historical television series, supplemented by offline real-life photos from a university Hanfu club.
- Curation: Images underwent a two-stage filtering process (coarse deduplication/text removal and fine-grained clarity checks).
- Annotation: Nine Hanfu experts annotated each outfit across 3 global features (Type, Period, Gender) and 5 local features (Sleeve, Collar, Jin/lapel, Bottoms, Outerwear). A feature value was only accepted if three annotators reached a uniform agreement.
Key Insights and Ideas
- The Multi-Image Paradox: Humans naturally perform better on multi-image tasks (MVQA) than single-image tasks (SVQA) because they can cross-reference and contrast features between images to deduce answers. VLMs exhibit the exact opposite behavior: their performance plummets on multi-image inputs, highlighting a severe bottleneck in cross-image relational reasoning.
- Language and Cultural Alignment Bias: VLMs perform significantly worse on SVQA when prompts are written in English rather than Chinese. This indicates that models struggle to align highly specific cultural and historical terminology across different languages.
- Background Context as a Double-Edged Sword: Keeping the original image background (rather than cropping it out) consistently improved VLM performance on global features like gender and garment type, yet it introduced distracting noise when models tried to identify fine-grained local features.
Key Results
The authors benchmarked state-of-the-art open-weights models (MiniCPM-V 2.6, Qwen2.5-VL-7B-Instruct, InternVL2.5) and closed-source models (GPT-4o, Doubao-1.5-V).
- Visual Understanding: The closed-source model Doubao-1.5-V achieved the highest overall accuracy (82.28% on SVQA and 75.95% on MVQA). Open-source models lagged significantly behind on multi-image tasks; for example, MiniCPM-V 2.6 dropped from 68.04% on SVQA to just 33.79% on MVQA.
- Human vs. Model: While closed VLMs perform comparably to non-expert humans on single-image tasks, they fall roughly 10% short of human experts.
- Transcreation Failure Rates: Human expert evaluation of the generated modern clothing designs showed that the best-performing generative model (Stable Diffusion v2.1) achieved a success rate of only 42% based on semantic consistency, modern adaptability, and cultural inheritance. Models frequently hallucinated or completely stripped away the defining historical elements.
Limitations and Future Work
The authors note several constraints in their current work:
- Scalability: The dataset is relatively small due to the strict requirement for unanimous expert consensus during annotation.
- Missing Metadata: The benchmark does not currently account for objective image variables like camera angles, lighting, or full- vs. half-body compositions, which can impact VLM performance.
- Data Leakage: Because some images were sourced from the internet, there is a minor risk that models encountered them during pre-training.
Future directions include developing fairness-aware, domain-specific fine-tuning methods for VLMs and fostering closer collaborations with cultural scholars to ensure AI-driven cultural preservation avoids inaccurate or offensive misrepresentations.
Summary updated