TL;DR
PSGAgent is a new agentic framework that transforms Multimodal Large Language Models (MLLMs) from static observers into active participants for Panoptic Scene Graph (PSG) generation. By using a "think–act–observe" loop with specialized visual tools, it overcomes the "perception bottleneck" of fixed-resolution views, achieving state-of-the-art performance even with a relatively small 3B parameter backbone.
What Problem Does This Paper Address?
Panoptic Scene Graph generation (PSG) is the task of mapping an image into a structured graph where nodes are objects and edges are their relationships. While MLLMs are great at reasoning, they struggle with a perception bottleneck when applied to PSG.
Standard models process images at a fixed resolution, leading to two major issues:
- Saliency Bias: Models focus on large, central objects and often miss critical relations involving small or peripheral ones (e.g., a person pushing a bicycle at the edge of a frame).
- Semantic Ambiguity: Irrelevant background noise can interfere with the model's attention, or subtle differences between actions (like "standing on" vs. "walking on") become blurred at low resolutions.
Methods
The authors propose PSGAgent, which treats scene graph generation as a multi-turn reasoning process. Instead of one-shot prediction, the agent follows a loop: it thinks about what it sees, acts by calling a tool to refine its view, and observes the new visual data.
The Agent's Toolbox
- Region Zoom-In Tool: Crops a specific area to help the model "see" small or marginal objects clearly.
- Visual Highlight Tool: Uses segmentation masks to black out everything except the subject and object of interest, filtering out contextual noise to resolve ambiguous predicates.
The Training Pipeline
Training an agent to use tools effectively is notoriously difficult due to sparse rewards. The authors use a two-stage approach:
- Cold-Start SFT: They created a high-quality Multimodal Chain-of-Thought (M-CoT) dataset by mining "hard" samples where standard models fail. This teaches the agent the basic logic of when to zoom or highlight.
- RL-based Enhancement: They use Group Sequence Policy Optimization (GSPO) with a hierarchical reward system. This rewards the agent not just for accuracy, but specifically for "causal" tool use—i.e., when a zoom action actually leads to discovering a previously missed relation.
Datasets
- PSG Dataset: The primary benchmark, containing ~48k images with 133 object and 56 predicate categories.
- Error-Aware M-CoT Dataset: A new dataset of 8,000 trajectories synthesized from challenging PSG samples to provide structured supervision for tool-use behavior.
Key Insights and Ideas
- Strategic Reasoning > Model Scale: One of the most striking findings is that a 3B parameter PSGAgent outperforms a vanilla 72B parameter model. This suggests that for complex visual tasks, how a model looks at data is more important than just having more parameters.
- Active Perception is Proactive: Unlike previous agents that only look for specific things when asked, PSGAgent is designed for "exhaustive discovery," meaning it proactively explores the scene to find all valid relations.
- Hierarchical Rewards Matter: Standard RL rewards (right/wrong) aren't enough for tool use. By adding specific rewards for "Causal Zooming" and "Strategic Highlighting," the authors stabilized training and improved the agent's efficiency.
Key Results
PSGAgent achieved State-of-the-Art (SOTA) results on the PSG benchmark:
- Recall@100: Reached 50.27%, significantly higher than previous MLLM-based methods like SGG-R³ (47.63%) and R1-SGG (43.48%).
- Fine-Grained Improvement: The agent showed massive gains on "bottleneck" sets—improving recall by +32.15% on saliency-biased samples and +30.87% on semantically ambiguous ones compared to a standard fine-tuned baseline.
- Qualitative Superiority: As shown in Figure 3, PSGAgent generates much denser and more logical graphs, avoiding the "hallucinations" (like a person standing on themselves) common in other models.
Limitations and Future Work
While highly effective, the framework currently relies on an external segmenter (like Mask2Former) for its Highlight tool. Although the authors showed the agent is robust to mask noise, a fully end-to-end trainable segmentation module within the agent could be even more powerful.
The authors suggest that this dynamic perception paradigm is a natural fit for Video Scene Graph Generation (VidSGG) and Embodied AI, where agents must constantly refine their visual understanding as they move through an environment.
Summary updated