Efficient Vision-Language pre-training via domain-specific learning for human activities

Adrian Bulat, Yassine Ouali, Ricardo Guerrero, Brais Martínez, Georgios TzimiropoulosEMNLP 2024 · DOI

Introduction

The prevailing trend in Vision-Language (VL) pre-training has been a relentless pursuit of scale, with models like CLIP and SigLIP training on billions of internet-scraped images to achieve broad generalist capabilities. While effective, this "more is better" approach is computationally expensive and often overkill for specialized applications such as sports analytics, security, or healthcare.

Researchers from Samsung AI and the Technical University of Iasi propose a more efficient alternative: HC-VL. By focusing pre-training on the specific domain of human activities, they demonstrate that a model can achieve superior performance on relevant tasks with a fraction of the data and compute required by generalist giants.

Methodology

The core of the HC-VL approach lies in aligning the pre-training process with a target domain without the need for expensive new data collection.

LLM-Guided Data Curation

Instead of manual labeling, the authors leveraged GPT-3.5 to generate a hierarchical taxonomy of human activities. This taxonomy includes approximately 9,700 keywords across six categories (e.g., Physical, Leisure, Domestic & Health). This structured knowledge was used to filter the massive LAION-5B dataset, resulting in the Human-Centric Image-Text (HC-IT) dataset containing 167 million high-relevance samples.

Caption Enhancement and Training

To address the noise inherent in web-crawled text, the team used BLIP-2 to generate synthetic, high-quality captions. A CLIP-score-based sampling strategy was then employed to dynamically choose between original and synthetic text during training, ensuring the model learns from the most descriptive data.

The architecture follows the standard CLIP design for images (HC-VL). For video tasks, HC-VL+ introduces lightweight temporal attention branches to a frozen image backbone, enabling motion understanding through "fast adaptation" without sacrificing spatial recognition.

Results and Efficiency

The most striking result of this domain-specific approach is its efficiency. HC-VL achieves state-of-the-art performance while using significantly fewer resources than its competitors.

Accuracy vs. Training Samples

As shown in the figure above, HC-VL (trained on 167M samples) outperforms models like SigLIP, which are trained on up to 10 billion samples. This represents a 60x reduction in data requirements and a training schedule that is 10x to 100x shorter than typical generalist models.

Detailed Performance

The model's effectiveness is validated across a wide range of benchmarks, including image/video retrieval and zero-shot action recognition.

Retrieval performance on Flickr30k and MS-COCO

In retrieval tasks (Table above), HC-VL consistently outperforms established baselines like OpenCLIP and MetaCLIP across different architectures (ViT-B/32 and ViT-B/16). Notably, in zero-shot action recognition across 16 datasets, HC-VL+ established a new state-of-the-art, improving performance by an average of +6%.

Conclusion

HC-VL demonstrates that "quality over quantity" is a viable and highly efficient path for Vision-Language models. By focusing on a specific domain—human activities—the researchers achieved superior accuracy with a fraction of the training data. This methodology offers a blueprint for creating specialized, high-performance "expert" models in other critical fields like autonomous driving or medical imaging, where efficiency and precision are paramount.

Reference: Bulat, A., Ouali, Y., Guerrero, R., Martinez, B., & Tzimiropoulos, G. (2024). Efficient Vision-Language pre-training via domain-specific learning for human activities. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Summary updated

BibTeX
@inproceedings{bulat2024efficient,
  author = {Adrian Bulat and Yassine Ouali and Ricardo Guerrero and Brais Martínez and Georgios Tzimiropoulos},
  title = {{Efficient Vision-Language pre-training via domain-specific learning for human activities}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.454},
}
Figure