Introduction
Multi-label image recognition (MLR) involves identifying multiple objects or attributes within a single image. While Vision-Language Models (VLMs) like CLIP have advanced this field, they often suffer from two primary limitations: the "plain template" problem and unshielded interaction. Standard models typically use generic prompts like "A photo of a {category}," failing to utilize the descriptive knowledge inherent in Large Language Models (LLMs). Furthermore, during the alignment of text labels with image patches, background noise often interferes with label representations, leading to suboptimal performance.
SSPA (Split-and-Synthesize Prompting with Gated Alignments) addresses these challenges by leveraging LLM-generated descriptions and a novel bidirectional gating mechanism to filter visual noise, achieving state-of-the-art performance across diverse datasets.
Methodology
The SSPA framework is built upon three innovative components designed to enhance label representation and visual-textual alignment.
Split-and-Synthesize Prompting (SSP)
SSPA generates two distinct types of prompts to capture comprehensive label information. Knowledge-Aware Prompting (KAP) queries LLMs for detailed physical descriptions (e.g., "A keyboard is a rectangular device with multiple keys"). Context-Aware Prompting (CAP) utilizes learnable tokens that adapt to the specific visual characteristics of a dataset. These sources are synthesized using Quaternion Semantic Modeling (QSM), which employs hyper-complex numbers to effectively model multi-dimensional correlations between modalities without significantly increasing parameter counts.
Gated Dual-Modal Alignments (GDMA)
To mitigate the impact of background noise, SSPA implements a bidirectional interaction mechanism. A Gating Mechanism acts as a mathematical filter, learning to suppress redundant information from irrelevant image patches while amplifying useful signals for specific labels. This ensures that the "conversation" between visual and textual features remains focused on relevant content.
Soft Aggregator
Instead of relying on a single global feature, SSPA employs a soft aggregator. This component evaluates all image regions and weights their importance, facilitating the recognition of small or partially obscured objects.
Experimental Results
The authors evaluated SSPA across nine datasets spanning natural images (MS-COCO, VOC 2007, NUS-WIDE), pedestrian attributes (PA100K, RAPv1, PETA), and remote sensing (MultiScene, MLRSNet, AID).
As shown in the results, SSPA achieved a new state-of-the-art on MS-COCO with 90.1% mAP using a ViT-B/16 backbone. It also demonstrated strong generalization in specialized domains like remote sensing, outperforming specialized models on the MultiScene dataset by over 1.2% mAP. Notably, SSPA is highly efficient; it outperforms the Query2Label (Q2L) model while utilizing approximately 65% fewer parameters (50.8M vs 143.1M).
Conclusion
SSPA demonstrates that LLMs can serve as effective "feature engineers" by providing structural information that aids VLMs in object localization. By combining detailed prompting with gated bidirectional alignments, the framework effectively filters noise and enhances multi-label recognition. Future work will focus on extending SSPA to open-vocabulary scenarios to further exploit the zero-shot capabilities of LLMs.
Summary updated