CogNet: Multi-Agent Collaborative Reasoning and Verification for Salient Object Ranking

Zhenyu Wu, Tengfei Shi, Xuehao Wang, Ming Li, Chenglizhao Chen, Wenfeng Song, Aimin HaoCVPR 2026 Findings 2026 · OpenReview

TL;DR

CogNet is a training-free, cognition-inspired framework for Salient Object Ranking (SOR) that mimics the human visual system by breaking the task into explicit perception, reasoning, and verification stages. By combining an instance-aware Segment Anything Model (SAM) with a multi-agent brainstorming mechanism using Multimodal Large Language Models (MLLMs), it achieves new state-of-the-art performance while remaining fully explainable.

What Problem Does This Paper Address?

Salient Object Ranking (SOR) is a computer vision task that requires a model to not only detect objects in an image but also rank them based on how much they naturally catch human attention.

The authors argue that current deep learning approaches for SOR suffer from two major flaws:

  1. Lack of a "Brain": Traditional models act as black boxes, directly mapping pixels to ranking scores without any transparent, step-by-step reasoning or verification.
  2. Perception Issues: Popular foundation models like SAM (Segment Anything Model) tend to over-segment single objects into multiple sub-patches, making them difficult to rank as cohesive instances.

To bridge this gap, CogNet introduces a paradigm shift that mirrors the human cognitive workflow: first perceiving the complete objects, then reasoning about their relative saliency, and finally verifying the results through collaborative discussion.

Figure 1. Highlights of CogNet. We introduces an entirely new learning paradigm that mimics human cognitive processes to per- form SOR, a capability absent in current deep SOR methods.

Methods

CogNet is entirely training-free and leverages off-the-shelf foundation models organized into three distinct stages:

Figure 1. Highlights of CogNet. We introduces an entirely new learning paradigm that mimics human cognitive processes to per- form SOR, a capability absent in current deep SOR methods.

1. Perception: Instance-Aware SAM (ISAM)

To overcome SAM's over-segmentation problem, ISAM assigns semantic labels to generated patches using a lightweight MLLM (Llama-3.2-1B). It then utilizes a novel Text-guided Graph Iterative Merging (TGIM) strategy. TGIM models the patches as nodes in a graph and iteratively merges adjacent patches that share the same semantic object description. This reduces the patch-merging computational complexity from a prohibitive $O(n^2)$ to a highly efficient $O(n)$.

2. Reasoning: Decomposed Chain-of-Thought (DCoT)

Standard Chain-of-Thought (CoT) prompting often struggles with complex visual ranking and introduces hallucinations. CogNet introduces DCoT, which breaks down the main ranking task into simpler sub-tasks:

  • Hierarchical Decomposition (HDCoT): Evaluates specific visual properties (like color brightness, object size, and distance to the image center) in parallel.
  • Recursive Decomposition (RDCoT): Iteratively asks the model to identify the most eye-catching object, mentally removes it from the scene, and repeats the process for the remaining objects.
Illustration of Hierarchical vs. Recursive Decomposed Chain-of-Thought.

3. Verification: Multi-Agent Brainstorming (MAB)

Instead of relying on a simple majority vote—which can fall into a "groupthink trap" if the majority is wrong—MAB introduces a collaborative discussion phase. Three distinct, lightweight MLLMs (InternVL2.5-1B, Llama-3.2-1B, and miniCPM-1B) act as agents. They present their preliminary rankings alongside estimated confidence scores, critique each other's reasoning, and update their answers over multiple rounds until a consensus is reached.

Datasets

The framework was evaluated on three widely accepted benchmark datasets for salient object ranking:

  • ASSR (Attention Shift Saliency Ranking)
  • IRSR (Instance Relative Saliency Ranking)
  • SIFR (Salient Instance Fixation Ranking)

Key Insights and Ideas

  • Cognitive Structuring Beats End-to-End Black Boxes: Explicitly separating perception, reasoning, and verification yields superior ranking accuracy and makes the model's decision-making process fully transparent and explainable to humans.
  • Alleviating the Groupthink Trap: Allowing diverse AI agents to debate and adjust their confidence levels dynamically prevents the ensemble from settling on common, superficial errors.
  • The Power of Zero-Shot Composition: CogNet proves that by carefully structuring the interaction between existing vision and language foundation models, you can outperform specialized models without fine-tuning a single weight.

Key Results

  • State-of-the-Art Performance: CogNet outperforms all existing state-of-the-art SOR methods, exceeding the second-best models by approximately 3.6% on the SA-SOR (Segmentation-Aware SOR) metric and 2.4% on the MAE (Mean Absolute Error) metric across all three datasets.
  • Efficiency vs. Accuracy Trade-off: The full CogNet framework runs at 14 FPS on an NVIDIA RTX 4090. A "slim" version of the model (which removes the multi-agent verification stage) slims the parameter footprint down to 854.3 million and runs at 32 FPS while still delivering highly competitive results.
  • Added Capabilities: Beyond traditional ranking, CogNet natively supports language-driven interactive segmentation and provides clear textual justifications for its rankings.

Limitations and Future Work

The primary limitation of CogNet is its computational overhead. Because it orchestrates multiple large language and vision models simultaneously, it possesses a significantly larger parameter footprint (up to 2.6x more parameters) than traditional, task-specific deep SOR networks.

In future work, the authors plan to address this by distilling the knowledge of CogNet into a single, lightweight network and designing more computationally efficient multi-agent verification mechanisms.

Summary updated

Abstract

Despite significant advancements in salient object ranking (SOR), current methods still fall short of matching the capabilities of the human visual system. We argue that the fundamental reason for this gap is that existing SOR approaches lack a brain that can perform analysis, reasoning, and verification like human. In this paper, we aim to bridge this gap by designing a cognition-inspired network (CogNet) that mimics the human labeling process, with three key innovations: instance-aware SAM (ISAM) for perception, decomposed chain-of-thought (DCoT) for reasoning, and multi-agent brainstorming (MAB) for verification. 1) ISAM serves as the ``eyes'' to perceive input images and segment complete object instances. 2) DCoT is designed to tackle complex reasoning tasks by decomposing chain-of-thought (CoT) prompts into simpler sub-tasks. 3) MAB introduces a multi-agent discussion and reflection mechanism to ensure the correctness of reasoning. Extensive experiments demonstrate that CogNet achieves new state-of-the-art performance on ASSR, IRSR, and SIFR datasets. The code will be publicly available at https://sites.google.com/view/sor-cognet.
BibTeX
@inproceedings{wu2026cognet,
  author = {Zhenyu Wu and Tengfei Shi and Xuehao Wang and Ming Li and Chenglizhao Chen and Wenfeng Song and Aimin Hao},
  title = {{CogNet: Multi-Agent Collaborative Reasoning and Verification for Salient Object Ranking}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=HzAOKyS29R},
}
Figure