WiCo: Win-win Cooperation of Bottom-up and Top-down Referring Image Segmentation

Zesen Cheng, Peng Jin, Hao Li, Kehan Li, Siheng Li, Xiangyang Ji, Chang Liu, Jie Chen

Introduction

Referring Image Segmentation (RIS) is a challenging multi-modal task that requires generating a precise pixel-level mask for an object described by a natural language expression. Despite significant progress, current models often struggle with either identifying the correct object in a complex scene or capturing the full extent of the target's geometry.

Existing RIS methodologies generally fall into two categories: Top-down (TD) and Bottom-up (BU). While both have distinct advantages, they also possess inherent flaws. This paper introduces WiCo (Win-win Cooperation), a framework designed to harmonize these two paradigms through an "Interaction then Integration" strategy, effectively mitigating the weaknesses of each to achieve state-of-the-art performance.

The Complementarity of TD and BU Approaches

To understand the motivation behind WiCo, it is essential to analyze the specific failure modes of existing strategies:

  1. Top-down (TD) Methods: These models typically utilize an object detector to generate candidate proposals and then select the one that best aligns with the text. While they excel at global context and object-level reasoning, they frequently suffer from Polar Negative (PN) errors—situations where the model selects the entirely wrong object (IoU $\approx$ 0) because it lacks the fine-grained detail to distinguish between similar instances.
  2. Bottom-up (BU) Methods: These models perform dense, pixel-wise alignment between the image and the text. While they are highly sensitive to detail, they often suffer from Inferior Positive (IP) errors. In these cases, the model identifies the correct object but produces an incomplete mask (IoU 0.5–0.8), missing parts like limbs or tools because it lacks a holistic "object-centric" understanding.

WiCo is built on the observation that these two approaches are naturally complementary: the global strength of TD can correct the fragmentation of BU, while the local precision of BU can resolve the ambiguity of TD.

The WiCo Framework

WiCo implements a dual-branch architecture that facilitates cooperation through two primary modules:

Complementary Feature Interaction (CFI)

Rather than treating the TD and BU branches as isolated pipelines, the CFI module enables bidirectional communication during the feature extraction phase. The TD branch provides "object-centric" priors to the BU branch, guiding it to perceive the object as a unified whole. Conversely, the BU branch shares fine-grained pixel information with the TD branch, providing the discriminative details necessary to separate the target from similar-looking distractors.

Gaussian Scoring Integration (GSI)

Simple averaging of outputs often leads to "harmful interference," where a poor prediction from one branch degrades a good prediction from the other. To solve this, WiCo employs Gaussian Scoring Integration. The framework models the performance of each branch as a Gaussian distribution, predicting both a mean and a standard deviation for the expected Intersection-over-Union (IoU). By sampling from these distributions, the model adaptively weights the contributions of each branch, placing higher trust in the branch that demonstrates greater confidence for a specific input.

Experimental Results and Insights

Performance Benchmarks

The authors evaluated WiCo on three major benchmarks: RefCOCO, RefCOCO+, and G-Ref (RefCOCOg). The framework consistently improved baseline models by 2.25% to 6.66% across all datasets. Notably, by combining advanced models such as LAVT (BU) and MAttNet (TD), WiCo achieved a state-of-the-art IoU of 75.50% on the RefCOCO validation set.

The Value of Heterogeneity

A key finding of the research is that combining a TD model with a BU model (heterogeneous) is significantly more effective than combining two models of the same type (homogeneous). The authors quantified this using the Mutually Exclusive Rate (MER), demonstrating that heterogeneous models are more likely to fail in different ways, thereby offering a higher potential for mutual correction.

Conclusion

WiCo demonstrates that the historical divide between top-down and bottom-up approaches in RIS is unnecessary. By fostering interaction and using probabilistic integration, the framework effectively eliminates Polar Negative and Inferior Positive errors. While the dual-branch approach increases computational overhead, the significant gains in accuracy and robustness suggest that "win-win cooperation" is a promising direction for multi-modal tasks, potentially extending to video grounding and tracking in the future.

Summary updated

BibTeX
@inproceedings{cheng2023wico,
  author = {Zesen Cheng and Peng Jin and Hao Li and Kehan Li and Siheng Li and Xiangyang Ji and Chang Liu and Jie Chen},
  title = {{WiCo: Win-win Cooperation of Bottom-up and Top-down Referring Image Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/71},
}
Figure