Introduction
The prevailing trend in Vision-Language (VL) pre-training has been a relentless pursuit of scale, with models like CLIP and SigLIP training on billions of internet-scraped images to achieve broad generalist capabilities. While effective, this "more is better" approach is computationally expensive and often overkill for specialized applications such as sports analytics, security, or healthcare.
Researchers from Samsung AI and the Technical University of Iasi propose a more efficient alternative: HC-VL. By focusing pre-training on the specific domain of human activities, they demonstrate that a model can achieve superior performance on relevant tasks with a fraction of the data and compute required by generalist giants.
Methodology
The core of the HC-VL approach lies in aligning the pre-training process with a target domain without the need for expensive new data collection.
LLM-Guided Data Curation
Instead of manual labeling, the authors leveraged GPT-3.5 to generate a hierarchical taxonomy of human activities. This taxonomy includes approximately 9,700 keywords across six categories (e.g., Physical, Leisure, Domestic & Health). This structured knowledge was used to filter the massive LAION-5B dataset, resulting in the Human-Centric Image-Text (HC-IT) dataset containing 167 million high-relevance samples.
Caption Enhancement and Training
To address the noise inherent in web-crawled text, the team used BLIP-2 to generate synthetic, high-quality captions. A CLIP-score-based sampling strategy was then employed to dynamically choose between original and synthetic text during training, ensuring the model learns from the most descriptive data.
The architecture follows the standard CLIP design for images (HC-VL). For video tasks, HC-VL+ introduces lightweight temporal attention branches to a frozen image backbone, enabling motion understanding through "fast adaptation" without sacrificing spatial recognition.
Results and Efficiency
The most striking result of this domain-specific approach is its efficiency. HC-VL achieves state-of-the-art performance while using significantly fewer resources than its competitors.
As shown in the figure above, HC-VL (trained on 167M samples) outperforms models like SigLIP, which are trained on up to 10 billion samples. This represents a 60x reduction in data requirements and a training schedule that is 10x to 100x shorter than typical generalist models.
Detailed Performance
The model's effectiveness is validated across a wide range of benchmarks, including image/video retrieval and zero-shot action recognition.
In retrieval tasks (Table above), HC-VL consistently outperforms established baselines like OpenCLIP and MetaCLIP across different architectures (ViT-B/32 and ViT-B/16). Notably, in zero-shot action recognition across 16 datasets, HC-VL+ established a new state-of-the-art, improving performance by an average of +6%.
Conclusion
HC-VL demonstrates that "quality over quantity" is a viable and highly efficient path for Vision-Language models. By focusing on a specific domain—human activities—the researchers achieved superior accuracy with a fraction of the training data. This methodology offers a blueprint for creating specialized, high-performance "expert" models in other critical fields like autonomous driving or medical imaging, where efficiency and precision are paramount.
Reference: Bulat, A., Ouali, Y., Guerrero, R., Martinez, B., & Tzimiropoulos, G. (2024). Efficient Vision-Language pre-training via domain-specific learning for human activities. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Summary updated