FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks

Xiao Han, Xiatian Zhu, Licheng Yu, Li Zhang, Yi-Zhe Song, Tao Xiang

Introduction

In the rapidly evolving field of fashion AI, Vision-and-Language (V+L) applications encompass a broad spectrum of tasks, from cross-modal retrieval and sub-category recognition to complex text-guided image retrieval and automated product captioning. These tasks are inherently heterogeneous, characterized by diverse input/output formats and significant variations in dataset scales. Traditionally, addressing these requirements necessitated training separate, large-scale models for each specific task—an approach that is computationally expensive, storage-intensive, and fails to leverage shared cross-task knowledge.

FAME-ViL (FAshion-focused Multi-task Efficient learning) addresses these challenges by introducing a unified architecture capable of performing multiple fashion-related tasks simultaneously. By employing a lightweight adapter-based design and a multi-teacher distillation strategy, FAME-ViL outperforms specialized single-task models while reducing total parameter requirements by over 60%.

Methodology

FAME-ViL is built upon a pre-trained CLIP (ViT-B/16) backbone. To maintain efficiency, the backbone remains largely frozen, with task-specific capabilities enabled through two types of lightweight adapters:

  1. Cross-Attention Adapter (XAA): While standard CLIP uses dual streams for images and text that only interact at the final stage, XAA introduces a bridge for deep modality fusion. This is critical for tasks requiring a nuanced understanding of both modalities, such as captioning and guided retrieval.
  2. Task-Specific Adapter (TSA): These parallel MLP layers absorb the unique characteristics of individual tasks, preventing interference with the shared knowledge stored in the backbone. The authors found that these two adapter types play complementary roles, with their combined use yielding performance gains greater than the sum of their individual contributions.

Operational Modes

The model dynamically switches between three operational modes to accommodate different task requirements:

  • Contrastive Mode: Optimized for simple cross-modal retrieval (matching images to text).
  • Fusion Mode: Designed for classification and text-guided retrieval, where image and text features must be blended.
  • Generative Mode: Configured as an encoder-decoder for fashion image captioning.

Multi-Teacher Distillation (MTD)

To mitigate "negative transfer"—where learning one task degrades performance on another—the authors utilize a Multi-Teacher Distillation strategy. Individual "expert" models are first trained for each task. The multi-task FAME-ViL model then acts as a student, learning to mimic these experts. This ensures the model balances learning across tasks of different scales and complexities without being overwhelmed by the largest datasets.

Experimental Setup

The model was evaluated on two major fashion benchmarks:

  • FashionGen: A large-scale dataset (~260k pairs) used for Cross-Modal Retrieval (XMR), Sub-Category Recognition (SCR), and Fashion Image Captioning (FIC).
  • FashionIQ: A dataset of ~18k training triplets used for Text-Guided Image Retrieval (TGIR), which requires modifying a reference image based on text instructions to find a target image.

Results and Discussion

FAME-ViL demonstrates superior performance across all evaluated tasks while maintaining high parameter efficiency.

Radar chart comparing FAME-ViL performance and parameter efficiency against baselines.

Efficiency and Performance

The model achieves a 61.5% reduction in parameters compared to maintaining four separate single-task models. Despite its smaller footprint, it significantly outperforms previous specialized models:

  • Text-Guided Retrieval (TGIR): Achieved a mean R@K of 58.29, a substantial improvement over the previous state-of-the-art (51.00).
  • Fashion Image Captioning (FIC): Reached a CIDEr score of 150.4, nearly five times the performance of earlier models like FashionBERT.

Qualitative Analysis

The model's versatility is further evidenced by its qualitative outputs in recognition and generation tasks, accurately identifying niche categories and generating descriptive, context-aware captions.

Qualitative examples of Sub-Category Recognition and Fashion Image Captioning outputs.

Conclusion and Future Work

FAME-ViL proves that a single, efficient model can master the heterogeneous landscape of fashion AI. While the current architecture uses simple addition-based fusion in XAA modules, future iterations could explore more sophisticated fusion mechanisms. Additionally, implementing adaptive algorithms to automatically tune adapter sizes could further optimize the balance between efficiency and performance. As it stands, FAME-ViL offers a robust, scalable solution for real-world e-commerce applications where multi-tasking and efficiency are paramount.

Summary updated

Abstract

In the fashion domain, there exists a variety of vision-and-language (V+L) tasks, including cross-modal retrieval, text-guided image retrieval, multi-modal classification, and image captioning. They differ drastically in each individual input/output format and dataset size. It has been common to design a task-specific model and fine-tune it independently from a pre-trained V+l model (e.g., CLIP). This results in parameter inefficiency and inability to exploit inter-task relatedness. To address such issues, we propose a novel FAshion-focused Multi-task Efficient learning method for Vision-and-Language tasks (FAME-ViL) in this work. Compared with existing approaches, FAME-ViL applies a single model for multiple heterogeneous fashion tasks, therefore being much more parameter-efficient. It is enabled by two novel components: (1) a task-versatile architecture with cross-attention adapters and task-specific adapters integrated into a unified V+L model, and (2) a stable and effective multi-task training strategy that supports learning from heterogeneous data and prevents negative transfer. Extensive experiments on four fashion tasks show that our FAME-ViL can save 61.5% of parameters over alternatives, while significantly outperforming the conventional independently trained single-task models. Code is available at https://github.com/BrandonHanx/FAME-ViL.
BibTeX
@inproceedings{han2023fame,
  author = {Xiao Han and Xiatian Zhu and Licheng Yu and Li Zhang and Yi-Zhe Song and Tao Xiang},
  title = {{FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure