Abstract
Multi-modal Large Language Models (MLLMs) frequently struggle to identify and segment small objects in high-resolution images due to the loss of spatial detail during image downsampling. We present FineRS, a novel two-stage reinforcement learning framework designed to bridge the gap between global reasoning and local perception. By employing a coarse-to-fine "thinking" process and a retrospective reward system, FineRS enables MLLMs to effectively locate, reason about, and segment ultra-small objects in 4K imagery.
1. Introduction
While state-of-the-art MLLMs like GPT-4V and LLaVA demonstrate impressive reasoning capabilities, they possess a significant "vision" bottleneck: high-resolution images must typically be downsampled to fit fixed-size neural architectures. In a 4K image, a small object—such as a specific vehicle in a dense parking lot—may be reduced to a few blurry pixels, rendering precise segmentation impossible. Existing solutions, such as patch-based processing, often suffer from high computational costs or a loss of global context. FineRS addresses these challenges by mimicking the human visual system's ability to scan a scene globally before zooming in on specific details.
2. Methodology
FineRS utilizes a two-stage pipeline that integrates reasoning with high-precision segmentation.
2.1 Global Semantic Exploration (GSE)
In the first stage, the model performs instruction-guided reasoning on the full high-resolution image. Utilizing a Chain-of-Thought (<think>) process, the model deliberates on spatial relationships and identifies a coarse target region (a 256x256 crop) where the object of interest is likely located. This allows the model to maintain global context while narrowing the search space.
2.2 Localized Perceptual Refinement (LPR)
The second stage focuses on the high-resolution crop identified by the GSE. The model performs "local reasoning" to generate an accurate bounding box and specific prompt points. These outputs are then fed into a frozen Segment Anything Model 2 (SAM2) to produce the final high-precision segmentation mask.
2.3 Reinforcement Learning via Retrospective Rewards
To synchronize these stages, FineRS employs Group Relative Policy Optimization (GRPO). The authors introduce a locate-informed retrospective reward system where the LPR stage acts as a critic for the GSE stage. If the LPR successfully segments the object within the selected crop, the GSE is rewarded; if the object is missing or truncated, the GSE is penalized. This feedback loop forces the global model to develop superior spatial awareness during its initial search.
3. The FINERS-4k Benchmark
To evaluate performance on ultra-small objects, the researchers introduced FINERS-4k, a benchmark consisting of 12,132 text-mask pairs derived from 4,563 high-resolution UAV (drone) images. The dataset categorizes objects into Small (S), Extra Small (XS), and Extra-Extra Small (XXS) tiers. It supports three primary tasks: Instruction-guided Segmentation (IS), Open-ended VQA (OVQA), and Multiple-choice VQA (MVQA).
4. Experimental Results
FineRS demonstrates significant performance gains over existing state-of-the-art models:
- Segmentation Accuracy: On the FINERS-4k test set, FineRS achieved a 55.1% gIoU (generalized Intersection over Union), vastly outperforming the LISA model (9.0%) and the previous RL-based Seg-Zero (32.1%).
- Reasoning Precision: In Multiple-choice VQA, the model reached 83.3% accuracy, confirming that its localization is driven by semantic understanding rather than simple pixel matching.
- Zero-Shot Generalization: When evaluated on the ReasonSeg dataset—which contains ground-level images distinct from the drone-based training data—FineRS outperformed models specifically fine-tuned for that domain, demonstrating the robustness of its coarse-to-fine strategy.
5. Discussion and Limitations
Despite its success, FineRS faces several challenges. The framework currently relies on a sequential dependency; if the GSE stage fails to locate the correct region, the LPR stage cannot recover the error. Additionally, the high memory requirements of running multiple large models simultaneously necessitate independent training of the stages rather than joint optimization. Finally, the two-stage "thinking" process introduces higher inference latency compared to single-pass models, presenting an opportunity for future architectural optimization.
6. Conclusion
FineRS represents a significant step forward in enabling MLLMs to handle complex, high-resolution visual tasks. By coupling global reasoning with localized refinement through reinforcement learning, the framework achieves state-of-the-art results in segmenting the smallest of objects, providing a scalable path for high-precision visual understanding.