LAVIS: A One-stop Library for Language-Vision Intelligence

Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, Steven C. H. HoiACL 2023 · DOI

Introduction

Language-Vision (LV) intelligence has seen rapid advancement, powering applications from automated captioning to multimodal search. However, this progress has resulted in a fragmented ecosystem where new state-of-the-art (SOTA) models often rely on bespoke codebases and inconsistent data-loading logic. This fragmentation creates a high barrier to entry for researchers and leads to significant repetitive effort. LAVIS (LAnguage-VISion) addresses these challenges by providing a centralized, extensible, and modular library designed to be a unified hub for the multimodal research community.

Modular Architecture

The core of LAVIS is a hierarchical design that decouples model and data definitions from training and evaluation logic. This structure is organized into six primary components:

  1. Runners: Manage the training and evaluation lifecycle, including checkpointing and iteration control.
  2. Tasks: Encapsulate logic for specific objectives such as retrieval, classification, or dialogue.
  3. Datasets: Handle data configurations and provide tools for automatic downloading of common public datasets.
  4. Models: Offer unified implementations of SOTA architectures like ALBEF, BLIP, BLIP-2, and CLIP.
  5. Processors: Transform raw multimodal inputs (images, video, text) into model-ready tensors.
  6. Common: Provide shared utilities for configuration, logging, and a registry system for easy extensibility.
The modular and hierarchical architecture of LAVIS, organizing components into tasks, datasets, models, processors, runners, and common utilities.

This unified interface allows users to load models and their corresponding preprocessors with minimal code, significantly streamlining the research pipeline.

Comprehensive Task and Dataset Support

LAVIS supports over 20 public datasets across image-text and video-text modalities, including MSCOCO, VQAv2, and LAION. The library is designed not just for training but for diverse inference tasks.

Demonstration of various language-vision tasks supported by LAVIS, including image description, zero-shot classification, text localization, and VQA.

By providing off-the-shelf inference capabilities, LAVIS enables practitioners to perform zero-shot classification or extract multimodal features with just a few lines of Python. The library also includes a GUI dataset browser to help researchers visually inspect data before training.

Key Results and Reproducibility

A primary goal of LAVIS is to ensure research reproducibility. The library's implementations faithfully replicate official results from original publications. For instance, BLIP on COCO achieves a Retrieval R@1 of 82.0 (Text-to-Image), matching official benchmarks. Furthermore, the framework has enabled new SOTA results; a fine-tuned BLIP model within LAVIS achieved 50.1% on the AOKVQA test split, surpassing previous specialized models.

Conclusion and Future Directions

LAVIS provides a robust foundation for the next generation of language-vision research. While currently comprehensive, future development will focus on expanding the model scope to include more specialized architectures and enhancing scalability for massive-scale parallelism. As these models are trained on large-scale web data, the authors also emphasize the importance of auditing for societal biases before deployment.

LAVIS is open-source and available on GitHub, inviting community contributions through its extensible registry system.

Summary updated

BibTeX
@inproceedings{li2023lavis,
  author = {Dongxu Li and Junnan Li and Hung Le and Guangsen Wang and Silvio Savarese and Steven C. H. Hoi},
  title = {{LAVIS: A One-stop Library for Language-Vision Intelligence}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-demo.3},
}
Figure