SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition

Hao Tan, Zichang Tan, Jun Li, Jun Wan, Zhen Lei, Stan Z. LiarXiv 2024 · DOI · arXiv

Introduction

Multi-label image recognition (MLR) involves identifying multiple objects or attributes within a single image. While Vision-Language Models (VLMs) like CLIP have advanced this field, they often suffer from two primary limitations: the "plain template" problem and unshielded interaction. Standard models typically use generic prompts like "A photo of a {category}," failing to utilize the descriptive knowledge inherent in Large Language Models (LLMs). Furthermore, during the alignment of text labels with image patches, background noise often interferes with label representations, leading to suboptimal performance.

SSPA (Split-and-Synthesize Prompting with Gated Alignments) addresses these challenges by leveraging LLM-generated descriptions and a novel bidirectional gating mechanism to filter visual noise, achieving state-of-the-art performance across diverse datasets.

Methodology

The SSPA framework is built upon three innovative components designed to enhance label representation and visual-textual alignment.

Overview of the SSPA framework architecture, illustrating the Split-and-Synthesize Prompting (SSP) and Gated Dual-Modal Alignments (GDMA) modules.

Split-and-Synthesize Prompting (SSP)

SSPA generates two distinct types of prompts to capture comprehensive label information. Knowledge-Aware Prompting (KAP) queries LLMs for detailed physical descriptions (e.g., "A keyboard is a rectangular device with multiple keys"). Context-Aware Prompting (CAP) utilizes learnable tokens that adapt to the specific visual characteristics of a dataset. These sources are synthesized using Quaternion Semantic Modeling (QSM), which employs hyper-complex numbers to effectively model multi-dimensional correlations between modalities without significantly increasing parameter counts.

Gated Dual-Modal Alignments (GDMA)

To mitigate the impact of background noise, SSPA implements a bidirectional interaction mechanism. A Gating Mechanism acts as a mathematical filter, learning to suppress redundant information from irrelevant image patches while amplifying useful signals for specific labels. This ensures that the "conversation" between visual and textual features remains focused on relevant content.

Soft Aggregator

Instead of relying on a single global feature, SSPA employs a soft aggregator. This component evaluates all image regions and weights their importance, facilitating the recognition of small or partially obscured objects.

Experimental Results

The authors evaluated SSPA across nine datasets spanning natural images (MS-COCO, VOC 2007, NUS-WIDE), pedestrian attributes (PA100K, RAPv1, PETA), and remote sensing (MultiScene, MLRSNet, AID).

Performance comparison on the MS-COCO dataset. SSPA achieves state-of-the-art results across various backbones and resolutions.

As shown in the results, SSPA achieved a new state-of-the-art on MS-COCO with 90.1% mAP using a ViT-B/16 backbone. It also demonstrated strong generalization in specialized domains like remote sensing, outperforming specialized models on the MultiScene dataset by over 1.2% mAP. Notably, SSPA is highly efficient; it outperforms the Query2Label (Q2L) model while utilizing approximately 65% fewer parameters (50.8M vs 143.1M).

Conclusion

SSPA demonstrates that LLMs can serve as effective "feature engineers" by providing structural information that aids VLMs in object localization. By combining detailed prompting with gated bidirectional alignments, the framework effectively filters noise and enhances multi-label recognition. Future work will focus on extending SSPA to open-vocabulary scenarios to further exploit the zero-shot capabilities of LLMs.

Summary updated

Abstract

Multi-label image recognition is a fundamental task in computer vision. Recently, Vision-Language Models (VLMs) have made notable advancements in this area. However, previous methods fail to effectively leverage the rich knowledge in language models and often incorporate label semantics into visual features unidirectionally. To overcome these problems, we propose a Split-and-Synthesize Prompting with Gated Alignments (SSPA) framework to amplify the potential of VLMs. Specifically, we develop an in-context learning approach to associate the inherent knowledge from LLMs. Then we propose a novel Split-and-Synthesize Prompting (SSP) strategy to first model the generic knowledge and downstream label semantics individually and then aggregate them carefully through the quaternion network. Moreover, we present Gated Dual-Modal Alignments (GDMA) to bidirectionally interact visual and linguistic modalities while eliminating redundant cross-modal information, enabling more efficient region-level alignments. Rather than making the final prediction by a sharp manner in previous works, we propose a soft aggregator to jointly consider results from all image regions. With the help of flexible prompting and gated alignments, SSPA is generalizable to specific domains. Extensive experiments on nine datasets from three domains (i.e., natural, pedestrian attributes and remote sensing) demonstrate the state-of-the-art performance of SSPA. Further analyses verify the effectiveness of SSP and the interpretability of GDMA. The code will be made public.
BibTeX
@misc{tan2024sspa,
  author = {Hao Tan and Zichang Tan and Jun Li and Jun Wan and Zhen Lei and Stan Z. Li},
  title = {{SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition}},
  year = {2024},
  eprint = {2407.20920},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.20920},
}
Figure