Abstract
The remarkable performance of large multimodal models (LMMs) has attracted significant interest from the image segmentation community. To align with the next-token-prediction paradigm, current LMM-driven segmentation methods either use object boundary points to represent masks or introduce special segmentation tokens, whose hidden states are decoded by a segmentation model requiring the original image as input. However, these approaches often suffer from inadequate mask representation and complex architectures, limiting the potential of LMMs. In this work, we propose the Hierarchical Mask Tokenizer (HiMTok), which represents segmentation masks with up to 32 tokens and eliminates the need for the original image during mask de-tokenization. HiMTok allows for compact and coarse-to-fine mask representations, aligning well with the LLM next-token-prediction paradigm and facilitating the direct acquisition of segmentation capabilities. We develop a 3-stage training recipe for progressive learning of segmentation and visual capabilities, featuring a hierarchical mask loss for effective coarse-to-fine learning. Additionally, we enable bidirectional information flow, allowing conversion between bounding boxes and mask tokens to fully leverage multi-task training potential. Extensive experiments demonstrate that our method achieves state-of-the-art performance across various segmentation tasks, while also enhancing visual grounding and maintaining overall visual understanding.
Introduction
While Large Multimodal Models (LMMs) have mastered text and basic image understanding, high-fidelity image segmentation remains a significant hurdle. Current approaches generally fall into two categories: polygon-based representations and hidden state-based methods. Polygon-based methods represent masks as sequences of boundary points, which often struggle with complex, non-convex, or multi-part shapes. Hidden state-based methods use the LLM to output a "special token" whose hidden state is fed into a heavy, external mask decoder like SAM. This latter approach requires re-processing the original image, leading to complex architectures and a disconnect between the LLM’s internal logic and the final mask.
HiMTok addresses these issues by introducing a compact, 1D token representation for masks that aligns perfectly with the standard next-token prediction paradigm, allowing the model to "speak" segmentation natively.
Methodology: The Hierarchical Mask Tokenizer
The core innovation of this work is HiMTok, which compresses a 2D binary mask into a sequence of just 32 hierarchical tokens. This approach treats masks as a "new language" with coarse-to-fine granularity.
Key Components
- Hierarchical Architecture: The tokenizer utilizes a causal attention mechanism where earlier tokens represent coarse location and global shape, while subsequent tokens refine fine-grained details.
- Hierarchical Mask Loss (HML): To enforce this structure, the model is supervised using Gaussian-blurred versions of the ground truth mask. For example, a 4-token sequence is trained to reconstruct a blurry "blob," while the full 32-token sequence aims for the sharp, original mask.
- Bidirectional Information Flow: The model is trained to convert bounding boxes to mask tokens and vice versa. This allows the mask to act as a form of "visual chain-of-thought," improving localization accuracy by asking the model to output mask tokens before bounding box coordinates.
3-Stage Training Recipe
- Mask Reconstruction: Training the HiMTok tokenizer/de-tokenizer in isolation to learn the mask "language."
- Joint Training: Integrating HiMTok with an LMM (InternVL2.5) using low-resolution images for efficiency.
- High-Res Fine-Tuning: Final polishing on high-resolution data to capture intricate details.
Experimental Results
HiMTok was evaluated across a massive collection of datasets, including ADE20K, COCO, and RefCOCO. The authors also introduced the Mask Perception (MaP) benchmark, a task requiring the model to choose the correct text description for a given mask, testing its ability to understand masks as input.
Performance Highlights
- Referring Expression Segmentation (RES): HiMTok achieved 85.0 cIoU on the RefCOCO validation set, significantly outperforming previous state-of-the-art models like LISA (74.9) and GLaMM (79.5).
- Reasoning Segmentation: On the ReasonSeg benchmark (e.g., "segment the thing the man is using to stay dry"), HiMTok reached 67.0 cIoU, setting a new record for models of its size.
- Efficiency: The mask representation is extremely compact, using only 32 tokens. Unlike previous methods, the HiMTok de-tokenizer does not need the original image to reconstruct the mask, making the inference pipeline much leaner.
Conclusion
HiMTok demonstrates that discretizing masks into a hierarchical sequence of tokens allows LMMs to leverage their autoregressive power for spatial reasoning. By eliminating the need for the original image during de-tokenization, HiMTok provides a leaner and more integrated approach to image segmentation. While the fixed token length of 32 is efficient, future work may explore adaptive token lengths to further optimize computation for simpler objects.
Summary updated