Abstract
HyperSeg is a universal segmentation assistant powered by a Visual Large Language Model (VLLM) designed for both image and video domains. It addresses the limitations of existing VLLM-based segmentation models, such as coarse visual features and multi-object confusion, by introducing a Fine-grained Visual Perceiver (FVP) and a Hybrid Entity Recognition (HER) strategy. HyperSeg achieves state-of-the-art performance across diverse tasks, including referring expression segmentation, panoptic segmentation, and complex video reasoning, outperforming significantly larger models.
Introduction
While Large Language Models (LLMs) have excelled in textual reasoning, applying this capability to pixel-level image segmentation remains challenging. Existing models often struggle with "blurry" vision due to low-resolution CLIP encoders, confusion in multi-object scenes, and a lack of temporal understanding for video tasks.
HyperSeg bridges these gaps by creating a unified assistant capable of handling a wide array of segmentation tasks through flexible text and visual prompts. As shown in Figure 1, the model masters everything from simple object selection to multi-step reasoning in dynamic video environments.
Methodology
The HyperSeg architecture integrates three primary innovations into a standard VLLM framework (e.g., Phi-2 or Vicuna) enhanced with LoRA for efficient tuning.
Fine-grained Visual Perceiver (FVP)
To overcome the limitations of single-scale CLIP features, HyperSeg employs a pyramid vision encoder. The FVP module utilizes conditional weighted cross-attention to fuse multi-scale, high-resolution features into the VLLM. This allows the model to capture fine boundaries and tiny details without overwhelming the language backbone.
Hybrid Entity Recognition (HER)
To improve reliability in complex scenes, HyperSeg adopts a two-stage recognition strategy. The VLLM is instructed to first generate object names (semantic recognition), which then enrich the "mask tokens" used by the final segmentation decoder. This hybrid approach combines generative reasoning with precise mask classification.
Temporal Adapter
For video segmentation, a temporal adapter module aggregates global prompts and injects local space-time information. This ensures temporal coherence, allowing the model to track objects accurately as they move or change across frames.
Experimental Results
HyperSeg was evaluated on a comprehensive suite of benchmarks covering image (COCO, RefCOCO) and video (DAVIS, ReVOS) segmentation.
- Referring Expression Segmentation: On RefCOCO+, HyperSeg achieved 79.0 cIoU, a 6.1-point improvement over the previous state-of-the-art. On gRefCOCO, it reached 47.5 cIoU.
- Video Reasoning: In complex video reasoning tasks (ReVOS), HyperSeg-3B achieved 53.0 J&F, significantly outperforming the 13B-parameter VISA model (40.9 J&F).
- Panoptic Segmentation: The model achieved 61.2 PQ on COCO, demonstrating that reasoning-centric models can excel at generic segmentation tasks.
Conclusion
HyperSeg demonstrates that high-resolution visual perception and explicit entity naming are critical for effective segmentation reasoning. While the multi-scale fusion adds some computational overhead, the model's efficiency and superior performance across image and video domains mark a significant step toward universal visual assistants. Future work will explore larger backbones and interactive editing capabilities.
Summary updated