Introduction
In the rapidly evolving field of fashion AI, Vision-and-Language (V+L) applications encompass a broad spectrum of tasks, from cross-modal retrieval and sub-category recognition to complex text-guided image retrieval and automated product captioning. These tasks are inherently heterogeneous, characterized by diverse input/output formats and significant variations in dataset scales. Traditionally, addressing these requirements necessitated training separate, large-scale models for each specific task—an approach that is computationally expensive, storage-intensive, and fails to leverage shared cross-task knowledge.
FAME-ViL (FAshion-focused Multi-task Efficient learning) addresses these challenges by introducing a unified architecture capable of performing multiple fashion-related tasks simultaneously. By employing a lightweight adapter-based design and a multi-teacher distillation strategy, FAME-ViL outperforms specialized single-task models while reducing total parameter requirements by over 60%.
Methodology
FAME-ViL is built upon a pre-trained CLIP (ViT-B/16) backbone. To maintain efficiency, the backbone remains largely frozen, with task-specific capabilities enabled through two types of lightweight adapters:
- Cross-Attention Adapter (XAA): While standard CLIP uses dual streams for images and text that only interact at the final stage, XAA introduces a bridge for deep modality fusion. This is critical for tasks requiring a nuanced understanding of both modalities, such as captioning and guided retrieval.
- Task-Specific Adapter (TSA): These parallel MLP layers absorb the unique characteristics of individual tasks, preventing interference with the shared knowledge stored in the backbone. The authors found that these two adapter types play complementary roles, with their combined use yielding performance gains greater than the sum of their individual contributions.
Operational Modes
The model dynamically switches between three operational modes to accommodate different task requirements:
- Contrastive Mode: Optimized for simple cross-modal retrieval (matching images to text).
- Fusion Mode: Designed for classification and text-guided retrieval, where image and text features must be blended.
- Generative Mode: Configured as an encoder-decoder for fashion image captioning.
Multi-Teacher Distillation (MTD)
To mitigate "negative transfer"—where learning one task degrades performance on another—the authors utilize a Multi-Teacher Distillation strategy. Individual "expert" models are first trained for each task. The multi-task FAME-ViL model then acts as a student, learning to mimic these experts. This ensures the model balances learning across tasks of different scales and complexities without being overwhelmed by the largest datasets.
Experimental Setup
The model was evaluated on two major fashion benchmarks:
- FashionGen: A large-scale dataset (~260k pairs) used for Cross-Modal Retrieval (XMR), Sub-Category Recognition (SCR), and Fashion Image Captioning (FIC).
- FashionIQ: A dataset of ~18k training triplets used for Text-Guided Image Retrieval (TGIR), which requires modifying a reference image based on text instructions to find a target image.
Results and Discussion
FAME-ViL demonstrates superior performance across all evaluated tasks while maintaining high parameter efficiency.
Efficiency and Performance
The model achieves a 61.5% reduction in parameters compared to maintaining four separate single-task models. Despite its smaller footprint, it significantly outperforms previous specialized models:
- Text-Guided Retrieval (TGIR): Achieved a mean R@K of 58.29, a substantial improvement over the previous state-of-the-art (51.00).
- Fashion Image Captioning (FIC): Reached a CIDEr score of 150.4, nearly five times the performance of earlier models like FashionBERT.
Qualitative Analysis
The model's versatility is further evidenced by its qualitative outputs in recognition and generation tasks, accurately identifying niche categories and generating descriptive, context-aware captions.
Conclusion and Future Work
FAME-ViL proves that a single, efficient model can master the heterogeneous landscape of fashion AI. While the current architecture uses simple addition-based fusion in XAA modules, future iterations could explore more sophisticated fusion mechanisms. Additionally, implementing adaptive algorithms to automatically tune adapter sizes could further optimize the balance between efficiency and performance. As it stands, FAME-ViL offers a robust, scalable solution for real-world e-commerce applications where multi-tasking and efficiency are paramount.
Summary updated