TL;DR
This paper introduces INTERN, a novel learning paradigm for general vision models that significantly reduces data requirements for new tasks. By employing a multi-stage pre-training approach and a comprehensive vision ecosystem, INTERN achieves state-of-the-art performance on numerous vision tasks, even when using only 10% of the original training data.
What Problem Does This Paper Address?
The current paradigm in computer vision often requires training a new model from scratch for each specific task, demanding vast amounts of well-annotated data and significant computational resources. This approach is inefficient and hinders the rapid adoption of AI across diverse scenarios, especially those with limited data. The paper addresses this challenge by proposing a new learning paradigm that aims to build a single, highly generalizable vision model capable of adapting to various tasks with minimal data.
Methods
INTERN introduces a three-stage upstream pre-training scheme:
- Upstream-Amateur (Up-A): Learns foundational skills from multi-modal (image-text, image-image, text-text) supervision signals. This stage has two phases: Up-A-G for global representation and Up-A-L for local representation, adapting models for dense prediction tasks.
- Upstream-Expert (Up-E): Specializes the model by accumulating knowledge from multiple data sources within specific task types (e.g., classification, detection, segmentation). This stage uses a "shared backbone" with dataset-specific heads.
- Upstream-Generalist (Up-G): Integrates knowledge from multiple experts into a unified model using a "hybrid parameter sharing" paradigm, creating a generalist model capable of handling diverse tasks.
Additionally, INTERN proposes a Downstream-Adaptation (Down-A) scheme using a Multi-stage Fine-tuning (MF) method, which leverages a VQ-GAN to reconstruct downstream data, decoupling it from upstream information and enabling efficient adaptation with limited data. The paper also introduces a new General Vision Architecture (GV-A) called MetaNet, a hybrid of convolutional and transformer blocks, and a General Vision Benchmark (GV-B) for comprehensive evaluation.
Datasets
The paper introduces a new General Vision Dataset (GV-D), a super-scale collection of vision datasets with 10 billion samples and various supervisory signals. GV-D comprises:
- GV-D-10B: 10 billion image-text pairs.
- GV-Dc-36M: 36 million images with classification labels from a 115K category label system.
- GV-Da-3M: 3 million images with 25 million bounding boxes for detection.
- GV-Ds-143K: 143 thousand images with segmentation masks.
The General Vision Benchmark (GV-B) includes 26 downstream tasks across four categories: classification, object detection, semantic segmentation, and depth estimation.
Key Insights and Ideas
- Multi-stage learning mimics human learning: The "amateur-expert-generalist" pre-training pipeline effectively absorbs knowledge from broad sources and scales efficiently.
- Diverse supervision is key for generality: Learning from multiple types of supervisory signals (image-level, bounding boxes, pixel-wise, natural language) is crucial for achieving true general vision capabilities.
- Data efficiency is paramount: INTERN models, adapted with only 10% of the training data, can outperform models trained with the full dataset, significantly reducing data dependency.
- Hybrid architectures offer a balance: MetaNet, a hybrid of convolutional and transformer blocks, achieves better generalization and efficiency.
- A unified vision ecosystem is beneficial: The combination of GV-D, GV-A, and GV-B fosters open research and development in general vision.
Key Results
- INTERN models, particularly Up-G MN-B15, achieve state-of-the-art performance on 26 well-known datasets, often outperforming models trained with 100% of the data when using only 10% of the data.
- For example, Up-G MN-B15 surpasses CLIP-R50x16 by +9.4% average accuracy on the classification suite, +11.4% AP on VOC detection, and +12.1% mIoU on semantic segmentation, all with 10% of the data.
- The MetaNet architecture family demonstrates strong performance and efficiency, with MN-B15 achieving 81.0% top-1 accuracy on ImageNet with 1.1 GFLOPS.
- The Downstream-Adaptation (Down-A) scheme significantly improves performance in few-shot settings, outperforming linear probe and fine-tuning methods.
Limitations and Future Work
- Generalist vs. Expert Performance: While the generalist model (Up-G) generally performs well, there are instances where individual experts (Up-E) outperform it on specific tasks, indicating that achieving uniformly stronger generalization than sub-nets is an ongoing research topic.
- Extensibility of Up-A: The generality of the Up-A stage on instance-level tasks requires further exploration, especially for dense and fine-grained tasks.
- Incorporating New Modalities: The authors suggest incorporating new modalities like video and audio, and exploring more complex cognitive tasks beyond current perception benchmarks.
- Ethical Considerations: The potential misuse of INTERN in surveillance and military applications is acknowledged, emphasizing the need for strict protocols and privacy protection in data handling.
- Data Privacy: The GV-D dataset may involve privacy issues (license plates, faces, documents), necessitating further research into privacy-preserving methods.
Summary updated