PSGAgent: Enhancing MLLMs' Dynamic Visual Perception for Panoptic Scene Graph Generation via Agentic Reasoning

Qingyun Liu, Yuankun Liu, Jiaye Feng, Jie Huang, Zonghong Dai, Haochen Li, Jingkun Wang, Weiping Li, Tong MoarXiv 2026 · DOI · arXiv

TL;DR

PSGAgent is a new agentic framework that transforms Multimodal Large Language Models (MLLMs) from static observers into active participants for Panoptic Scene Graph (PSG) generation. By using a "think–act–observe" loop with specialized visual tools, it overcomes the "perception bottleneck" of fixed-resolution views, achieving state-of-the-art performance even with a relatively small 3B parameter backbone.

What Problem Does This Paper Address?

Panoptic Scene Graph generation (PSG) is the task of mapping an image into a structured graph where nodes are objects and edges are their relationships. While MLLMs are great at reasoning, they struggle with a perception bottleneck when applied to PSG.

Standard models process images at a fixed resolution, leading to two major issues:

  1. Saliency Bias: Models focus on large, central objects and often miss critical relations involving small or peripheral ones (e.g., a person pushing a bicycle at the edge of a frame).
  2. Semantic Ambiguity: Irrelevant background noise can interfere with the model's attention, or subtle differences between actions (like "standing on" vs. "walking on") become blurred at low resolutions.
Figure 1: The perception bottleneck in PSG. (a) Saliency bias: static models are biased toward dominant regions, while ne- glecting small-scale or off-center relations. (b) Semantic am- biguity: long-range contextual interference and fine-grained predicate confusion lead to relational uncertainty.

Methods

The authors propose PSGAgent, which treats scene graph generation as a multi-turn reasoning process. Instead of one-shot prediction, the agent follows a loop: it thinks about what it sees, acts by calling a tool to refine its view, and observes the new visual data.

The Agent's Toolbox

  • Region Zoom-In Tool: Crops a specific area to help the model "see" small or marginal objects clearly.
  • Visual Highlight Tool: Uses segmentation masks to black out everything except the subject and object of interest, filtering out contextual noise to resolve ambiguous predicates.

The Training Pipeline

Training an agent to use tools effectively is notoriously difficult due to sparse rewards. The authors use a two-stage approach:

  1. Cold-Start SFT: They created a high-quality Multimodal Chain-of-Thought (M-CoT) dataset by mining "hard" samples where standard models fail. This teaches the agent the basic logic of when to zoom or highlight.
  2. RL-based Enhancement: They use Group Sequence Policy Optimization (GSPO) with a hierarchical reward system. This rewards the agent not just for accuracy, but specifically for "causal" tool use—i.e., when a zoom action actually leads to discovering a previously missed relation.
The two-stage training pipeline involving SFT and RL enhancement.

Datasets

  • PSG Dataset: The primary benchmark, containing ~48k images with 133 object and 56 predicate categories.
  • Error-Aware M-CoT Dataset: A new dataset of 8,000 trajectories synthesized from challenging PSG samples to provide structured supervision for tool-use behavior.

Key Insights and Ideas

  • Strategic Reasoning > Model Scale: One of the most striking findings is that a 3B parameter PSGAgent outperforms a vanilla 72B parameter model. This suggests that for complex visual tasks, how a model looks at data is more important than just having more parameters.
  • Active Perception is Proactive: Unlike previous agents that only look for specific things when asked, PSGAgent is designed for "exhaustive discovery," meaning it proactively explores the scene to find all valid relations.
  • Hierarchical Rewards Matter: Standard RL rewards (right/wrong) aren't enough for tool use. By adding specific rewards for "Causal Zooming" and "Strategic Highlighting," the authors stabilized training and improved the agent's efficiency.

Key Results

PSGAgent achieved State-of-the-Art (SOTA) results on the PSG benchmark:

  • Recall@100: Reached 50.27%, significantly higher than previous MLLM-based methods like SGG-R³ (47.63%) and R1-SGG (43.48%).
  • Fine-Grained Improvement: The agent showed massive gains on "bottleneck" sets—improving recall by +32.15% on saliency-biased samples and +30.87% on semantically ambiguous ones compared to a standard fine-tuned baseline.
  • Qualitative Superiority: As shown in Figure 3, PSGAgent generates much denser and more logical graphs, avoiding the "hallucinations" (like a person standing on themselves) common in other models.
Qualitative comparison showing PSGAgent's ability to find subtle relations and resolve ambiguity.

Limitations and Future Work

While highly effective, the framework currently relies on an external segmenter (like Mask2Former) for its Highlight tool. Although the authors showed the agent is robust to mask noise, a fully end-to-end trainable segmentation module within the agent could be even more powerful.

The authors suggest that this dynamic perception paradigm is a natural fit for Video Scene Graph Generation (VidSGG) and Embodied AI, where agents must constantly refine their visual understanding as they move through an environment.

Summary updated

Abstract

Panoptic Scene Graph generation (PSG) requires holistic yet fine-grained reasoning over complex relations among objects. However, current Multimodal Large Language Models (MLLMs) applied to PSG suffer from a perception bottleneck: they function as static observers limited by fixed-resolution views. As a result, models often overlook critical relations involving small or peripheral objects and struggle to differentiate ambiguous predicates in complex scenes. To address these limitations, we propose PSGAgent, an agentic framework that enhances visual perception for PSG through a think-act-observe loop. PSGAgent allows MLLMs to dynamically refine their visual perception using two tools: a Region Zoom-In Tool for local exploration of small or boundary relations and a Visual Highlight Tool for relation-focused segmentation that filters out irrelevant contextual noise. To optimize decision-making, we construct a two-stage training pipeline. First, we create a high-quality, error-aware Multimodal Chain-of-Thought (M-CoT) dataset that provides robust supervision for cold-start learning. Then, we design a novel hierarchical reward mechanism integrated with Group Sequence Policy Optimization (GSPO) to enable strategic and adaptive tool usage. Extensive experiments on the PSG benchmark show that PSGAgent achieves SOTA performance, demonstrating strong capability in both holistic and fine-grained scene understanding.
BibTeX
@misc{liu2026psgagent,
  author = {Qingyun Liu and Yuankun Liu and Jiaye Feng and Jie Huang and Zonghong Dai and Haochen Li and Jingkun Wang and Weiping Li and Tong Mo},
  title = {{PSGAgent: Enhancing MLLMs' Dynamic Visual Perception for Panoptic Scene Graph Generation via Agentic Reasoning}},
  year = {2026},
  eprint = {2611.07408},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07408},
}
Figure