Introduction
In the evolving landscape of computer vision, models have traditionally been specialized: some excel at segmenting pixels (e.g., SAM), others at matching images to text (e.g., CLIP), and many more are dedicated solely to video tracking. This fragmentation necessitates complex pipelines for multi-faceted tasks. GLEE (General Object Foundation Model) addresses this by unifying detection, segmentation, tracking, grounding, and identification into a single framework capable of zero-shot generalization across both images and videos.
Methods and Architecture
GLEE’s architecture is designed to be a "Swiss Army Knife" for object perception. It consists of four primary components: an image encoder for multi-scale feature extraction, a CLIP-based text encoder for processing natural language descriptions, a visual prompter for interactive inputs (points, boxes, scribbles), and an object decoder based on MaskDINO.
The core innovation is a unified training paradigm that treats all tasks as "object extraction." A dynamic class head computes similarity between object embeddings and multi-modal prompt embeddings. For video, GLEE employs contrastive learning to maintain object identity across frames, enabling tracking without specialized modules.
Data Scaling
GLEE's robust performance stems from a three-stage training strategy:
- Pre-training: Utilizing Objects365 and OpenImages for fundamental detection.
- Joint Training: Incorporating diverse datasets like COCO, LVIS, RefCOCO, and video benchmarks (YTVIS, OVIS).
- Scaling: Integrating 10 million images, including automatically labeled data from SA1B and GRIT, to enhance open-world capabilities.
Key Results
GLEE demonstrates exceptional versatility and performance. In zero-shot video tracking, it outperformed the previous state-of-the-art (OVTrack) by 36% on the TAO benchmark and nearly tripled performance on BURST.
As shown in the radar chart, GLEE achieves competitive or superior results compared to specialized models like UNINEXT and Florence-2 across a wide array of tasks, including Open-Vocabulary (OV) detection and video instance segmentation (VIS). Its performance on "Object Detection in the Wild" (ODinW) highlights the effectiveness of scaling with automatically labeled data.
Discussion and Conclusion
GLEE successfully bridges the semantic gap between "what" an object is and "where" it is. While it faces challenges in extreme occlusion scenarios and resolution sensitivity, its ability to generalize to open-world scenarios positions it as a powerful foundation for future multimodal Large Language Models (mLLMs). By treating video as a temporal extension of images, GLEE simplifies the vision pipeline into a single, scalable architecture.
Summary updated