INTERN: A New Learning Paradigm Towards General Vision

Jing Shao, Siyu Chen, Yangguang Li, Kun Wang, Zhenfei Yin, Yinan He, Jianing Teng, Qinghong Sun, Mengya Gao, Jihao Liu, Gengshi Huang, Guanglu Song, Yichao Wu, Yuming Huang, Fenggang Liu, Huan Peng, Shuo Qin, Chengyu Wang, Yujie Wang, Conghui He, Ding Liang, Yu Liu, Fengwei Yu, Junjie Yan, Dahua Lin, Xiaogang Wang, Yu QiaoarXiv 2021 · OpenReview

TL;DR

This paper introduces INTERN, a novel learning paradigm for general vision models that significantly reduces data requirements for new tasks. By employing a multi-stage pre-training approach and a comprehensive vision ecosystem, INTERN achieves state-of-the-art performance on numerous vision tasks, even when using only 10% of the original training data.

What Problem Does This Paper Address?

The current paradigm in computer vision often requires training a new model from scratch for each specific task, demanding vast amounts of well-annotated data and significant computational resources. This approach is inefficient and hinders the rapid adoption of AI across diverse scenarios, especially those with limited data. The paper addresses this challenge by proposing a new learning paradigm that aims to build a single, highly generalizable vision model capable of adapting to various tasks with minimal data.

Methods

INTERN introduces a three-stage upstream pre-training scheme:

  1. Upstream-Amateur (Up-A): Learns foundational skills from multi-modal (image-text, image-image, text-text) supervision signals. This stage has two phases: Up-A-G for global representation and Up-A-L for local representation, adapting models for dense prediction tasks.
  2. Upstream-Expert (Up-E): Specializes the model by accumulating knowledge from multiple data sources within specific task types (e.g., classification, detection, segmentation). This stage uses a "shared backbone" with dataset-specific heads.
  3. Upstream-Generalist (Up-G): Integrates knowledge from multiple experts into a unified model using a "hybrid parameter sharing" paradigm, creating a generalist model capable of handling diverse tasks.

Additionally, INTERN proposes a Downstream-Adaptation (Down-A) scheme using a Multi-stage Fine-tuning (MF) method, which leverages a VQ-GAN to reconstruct downstream data, decoupling it from upstream information and enabling efficient adaptation with limited data. The paper also introduces a new General Vision Architecture (GV-A) called MetaNet, a hybrid of convolutional and transformer blocks, and a General Vision Benchmark (GV-B) for comprehensive evaluation.

Datasets

The paper introduces a new General Vision Dataset (GV-D), a super-scale collection of vision datasets with 10 billion samples and various supervisory signals. GV-D comprises:

  • GV-D-10B: 10 billion image-text pairs.
  • GV-Dc-36M: 36 million images with classification labels from a 115K category label system.
  • GV-Da-3M: 3 million images with 25 million bounding boxes for detection.
  • GV-Ds-143K: 143 thousand images with segmentation masks.

The General Vision Benchmark (GV-B) includes 26 downstream tasks across four categories: classification, object detection, semantic segmentation, and depth estimation.

Key Insights and Ideas

  • Multi-stage learning mimics human learning: The "amateur-expert-generalist" pre-training pipeline effectively absorbs knowledge from broad sources and scales efficiently.
  • Diverse supervision is key for generality: Learning from multiple types of supervisory signals (image-level, bounding boxes, pixel-wise, natural language) is crucial for achieving true general vision capabilities.
  • Data efficiency is paramount: INTERN models, adapted with only 10% of the training data, can outperform models trained with the full dataset, significantly reducing data dependency.
  • Hybrid architectures offer a balance: MetaNet, a hybrid of convolutional and transformer blocks, achieves better generalization and efficiency.
  • A unified vision ecosystem is beneficial: The combination of GV-D, GV-A, and GV-B fosters open research and development in general vision.

Key Results

  • INTERN models, particularly Up-G MN-B15, achieve state-of-the-art performance on 26 well-known datasets, often outperforming models trained with 100% of the data when using only 10% of the data.
  • For example, Up-G MN-B15 surpasses CLIP-R50x16 by +9.4% average accuracy on the classification suite, +11.4% AP on VOC detection, and +12.1% mIoU on semantic segmentation, all with 10% of the data.
  • The MetaNet architecture family demonstrates strong performance and efficiency, with MN-B15 achieving 81.0% top-1 accuracy on ImageNet with 1.1 GFLOPS.
  • The Downstream-Adaptation (Down-A) scheme significantly improves performance in few-shot settings, outperforming linear probe and fine-tuning methods.

Limitations and Future Work

  • Generalist vs. Expert Performance: While the generalist model (Up-G) generally performs well, there are instances where individual experts (Up-E) outperform it on specific tasks, indicating that achieving uniformly stronger generalization than sub-nets is an ongoing research topic.
  • Extensibility of Up-A: The generality of the Up-A stage on instance-level tasks requires further exploration, especially for dense and fine-grained tasks.
  • Incorporating New Modalities: The authors suggest incorporating new modalities like video and audio, and exploring more complex cognitive tasks beyond current perception benchmarks.
  • Ethical Considerations: The potential misuse of INTERN in surveillance and military applications is acknowledged, emphasizing the need for strict protocols and privacy protection in data handling.
  • Data Privacy: The GV-D dataset may involve privacy issues (license plates, faces, documents), necessitating further research into privacy-preserving methods.

Summary updated

Abstract

Enormous waves of technological innovations over the past several years, marked by the advances in AI technologies, are profoundly reshaping the industry and the society. However, down the road, a key challenge awaits us, that is, our capability of meeting rapidly-growing scenario-specific demands is severely limited by the cost of acquiring a commensurate amount of training data. This difficult situation is in essence due to limitations of the mainstream learning paradigm: we need to train a new model for each new scenario, based on a large quantity of well-annotated data and commonly from scratch. In tackling this fundamental problem, we move beyond and develop a new learning paradigm named INTERN. By learning with supervisory signals from multiple sources in multiple stages, the model being trained will develop strong generalizability. We evaluate our model on 26 well-known datasets that cover four categories of tasks in computer vision. In most cases, our models, adapted with only 10% of the training data in the target domain, outperform the counterparts trained with the full set of data, often by a significant margin. This is an important step towards a promising prospect where such a model with general vision capability can dramatically reduce our reliance on data, thus expediting the adoption of AI technologies. Furthermore, revolving around our new paradigm, we also introduce a new data system, a new architecture, and a new benchmark, which, together, form a general vision ecosystem to support its future development in an open and inclusive manner. See project website at https://opengvlab.shlab.org.cn .
BibTeX
@misc{shao2021intern,
  author = {Jing Shao and Siyu Chen and Yangguang Li and Kun Wang and Zhenfei Yin and Yinan He and Jianing Teng and Qinghong Sun and Mengya Gao and Jihao Liu and Gengshi Huang and Guanglu Song and Yichao Wu and Yuming Huang and Fenggang Liu and Huan Peng and Shuo Qin and Chengyu Wang and Yujie Wang and Conghui He and Ding Liang and Yu Liu and Fengwei Yu and Junjie Yan and Dahua Lin and Xiaogang Wang and Yu Qiao},
  title = {{INTERN: A New Learning Paradigm Towards General Vision}},
  year = {2021},
  eprint = {2111.08687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.08687},
}
Figure