Abstract
Visual attribute learning at scale is often limited by partial labeling, where only a small fraction of potential attributes are annotated for each image. This paper introduces a vision-language assisted framework that leverages the zero-shot capabilities of CLIP to estimate the likelihood of missing labels. By employing a selective loss strategy that identifies and ignores ambiguous attributes during training, the proposed method avoids the pitfalls of traditional "ignoring" or "negatifying" strategies. Evaluations on the VAW dataset demonstrate state-of-the-art performance, achieving a mean Average Precision (mAP) of 64.06.
Introduction
Visual attributes (e.g., "furry," "striped," "metallic") are essential for fine-grained object understanding. However, labeling these attributes at scale is labor-intensive. In the VAW (Visual Attributes in the Wild) dataset, for instance, objects have an average of only 3.56 annotated attributes out of 620 possibilities. This leads to a "partial annotation" problem, where many present attributes are left unannotated, creating ambiguity for traditional training methods.
Standard approaches typically either ignore unannotated attributes, losing valuable data, or treat them all as negative, which introduces false negatives. Previous attempts to bridge these methods often fall into a "chicken-and-egg" trap, requiring a high-quality model to predict labels that are themselves needed to train the model.
Methodology
The proposed framework utilizes a pre-trained vision-language model (CLIP) as an external knowledge source to break the dependency cycle. The pipeline consists of three primary stages:
- Feasibility Filtering: The model identifies a "feasible set" ($\Omega_o$) of attributes for each object category based on co-occurrence in training data to reduce noise.
- Vision-Language Estimation: CLIP estimates the probability ($G_{x,o}$) of unannotated attributes being present by comparing image features to text prompts like "a photo of {attribute} {object}."
- Selective Loss: High-probability unannotated attributes are treated as ambiguous. During training, a subset of these is randomly selected to be ignored (zero loss), while the remainder are treated as negative. This randomized approach provides a robust training signal and acts as a form of regularization.
Experiments and Results
The study utilizes a cleaned version of the VAW dataset, comprising 170,407 visual instances across 1,763 object categories and 591 attributes. The proposed method achieved 64.06 mAP, significantly outperforming existing state-of-the-art methods like PSL (62.28 mAP) and ASL-negatifying (62.01 mAP).
| Method | Overall mAP |
|---|---|
| CE-ignoring | 61.18 |
| ASL-negatifying | 62.01 |
| PSL | 62.28 |
| Ours | 64.06 |
Qualitative results demonstrate that the model is capable of "filling in the blanks," correctly identifying attributes that were present in the scene but omitted by human annotators.
Discussion and Limitations
Performance is sensitive to the NUM_IGNORE hyperparameter, which determines the number of ambiguous attributes to skip. Ignoring all attributes leads to a significant drop in performance (59.41 mAP), highlighting the need for a balanced negative signal. Furthermore, the method's success is tied to the quality and potential biases of the underlying vision-language model used for estimation.
Conclusion
By leveraging CLIP to identify label uncertainty, this vision-language assisted approach effectively addresses the partial labeling problem in attribute learning. The selective loss strategy provides a scalable way to utilize unannotated data, setting a new benchmark for fine-grained visual recognition.
Summary updated