General Object Foundation Model for Images and Videos at Scale

Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, Song Bai

Introduction

In the evolving landscape of computer vision, models have traditionally been specialized: some excel at segmenting pixels (e.g., SAM), others at matching images to text (e.g., CLIP), and many more are dedicated solely to video tracking. This fragmentation necessitates complex pipelines for multi-faceted tasks. GLEE (General Object Foundation Model) addresses this by unifying detection, segmentation, tracking, grounding, and identification into a single framework capable of zero-shot generalization across both images and videos.

Methods and Architecture

GLEE’s architecture is designed to be a "Swiss Army Knife" for object perception. It consists of four primary components: an image encoder for multi-scale feature extraction, a CLIP-based text encoder for processing natural language descriptions, a visual prompter for interactive inputs (points, boxes, scribbles), and an object decoder based on MaskDINO.

The core innovation is a unified training paradigm that treats all tasks as "object extraction." A dynamic class head computes similarity between object embeddings and multi-modal prompt embeddings. For video, GLEE employs contrastive learning to maintain object identity across frames, enabling tracking without specialized modules.

GLEE unifies diverse vision tasks through multi-modal prompts and massive-scale data integration.

Data Scaling

GLEE's robust performance stems from a three-stage training strategy:

  1. Pre-training: Utilizing Objects365 and OpenImages for fundamental detection.
  2. Joint Training: Incorporating diverse datasets like COCO, LVIS, RefCOCO, and video benchmarks (YTVIS, OVIS).
  3. Scaling: Integrating 10 million images, including automatically labeled data from SA1B and GRIT, to enhance open-world capabilities.

Key Results

GLEE demonstrates exceptional versatility and performance. In zero-shot video tracking, it outperformed the previous state-of-the-art (OVTrack) by 36% on the TAO benchmark and nearly tripled performance on BURST.

Radar chart comparing GLEE against previous SOTA and specialized models across twelve diverse vision benchmarks.

As shown in the radar chart, GLEE achieves competitive or superior results compared to specialized models like UNINEXT and Florence-2 across a wide array of tasks, including Open-Vocabulary (OV) detection and video instance segmentation (VIS). Its performance on "Object Detection in the Wild" (ODinW) highlights the effectiveness of scaling with automatically labeled data.

Discussion and Conclusion

GLEE successfully bridges the semantic gap between "what" an object is and "where" it is. While it faces challenges in extreme occlusion scenarios and resolution sensitivity, its ability to generalize to open-world scenarios positions it as a powerful foundation for future multimodal Large Language Models (mLLMs). By treating video as a temporal extension of images, GLEE simplifies the vision pipeline into a single, scalable architecture.

Summary updated

Abstract

We present GLEE in this work, an object-level foundation model for locating and identifying objects in images and videos. Through a unified framework, GLEE accomplishes detection, segmentation, tracking, grounding, and identification of arbitrary objects in the open world scenario for various object perception tasks. Adopting a cohesive learning strategy, GLEE acquires knowledge from diverse data sources with varying supervision levels to formu-late general object representations, excelling in zero-shot transfer to new data and tasks. Specifically, we employ an image encoder, text encoder, and visual prompter to handle multimodal inputs, enabling to simultaneously solve various object-centric downstream tasks while maintaining state-of-the-art performance. Demonstrated through extensive training on over five million images from diverse benchmarks, GLEE exhibits remarkable versatility and improved generalization performance, efficiently tack-ling downstream tasks without the need for task-specific adaptation. By integrating large volumes of automatically labeled data, we further enhance its zero-shot generalization capabilities. Additionally, GLEE is capable of being integrated into Large Language Models, serving as a foun-dational model to provide universal object-level information for multimodal tasks. We hope that the versatility and universality of our method will mark a significant step in the development of efficient visual foundation models for AGI systems. The models and code are released at https://github.com/FoundationVision/GLEE.
BibTeX
@misc{wu2023general,
  author = {Junfeng Wu and Yi Jiang and Qihao Liu and Zehuan Yuan and Xiang Bai and Song Bai},
  title = {{General Object Foundation Model for Images and Videos at Scale}},
  year = {2023},
  eprint = {2312.09158},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.09158},
}
Figure