Abstract
This paper presents a framework for object recognition that utilizes semantic attributes as "hidden information" available exclusively during the training phase. Unlike traditional sequential models that suffer from error propagation due to unreliable attribute detection at test time, the proposed approach uses attributes to regularize the learning process. By formulating the problem within the Learning with Hidden Information (LHI) paradigm, we develop models that leverage object-attribute relationships to structure the latent space of the classifier. Experimental results on the aPascal and Animals with Attributes (AWA) datasets demonstrate that our combined model, xLR-Rel+, consistently outperforms baseline and state-of-the-art attribute-based methods while requiring only standard image features during inference.
Introduction
In computer vision, attribute-based recognition typically follows a two-step sequential process: first, a model predicts attributes from an image (e.g., "has wings," "is furry"); second, these predicted attributes are used to identify the object category. However, attribute recognition is inherently difficult. If an intermediate attribute detector fails, the final object classifier is likely to fail as well. This "error propagation" means that poor-quality middle-level representations often degrade final performance.
This research addresses this bottleneck by asking: Can we use the rich semantic information of attributes to help a model learn better features during training, without requiring those attributes during the testing phase?
Methodology
The authors propose a framework based on the Learning with Hidden Information (LHI) paradigm. The core idea is to use attributes as "privileged information" that guides the training of a standard image-based classifier.
The study introduces three primary formulations:
- xLR+ (Attributes as Extra Information): This approach trains two models simultaneously: a "target model" (using image features) and a "hypothetic model" (using ground-truth attributes). The training objective minimizes the discrepancy between the target model's predictions and the hypothetic model's predictions, effectively using attributes as a "teacher."
- LR-Rel+ (Object-Attribute Relationships): This method exploits known semantic links between objects and attributes (e.g., "birds" are statistically likely to have "beaks"). It ensures the classifier's output preserves these known relationships.
- xLR-Rel+ (The Combined Model): This final formulation integrates both hypothetic model regularization and relationship preservation into a single convex objective function.
Crucially, because these attributes are treated as hidden information during training, the resulting classifier only requires standard image features (like HOG or SIFT) during testing, avoiding the risk of misidentifying attributes at runtime.
Experimental Results
The proposed models were evaluated on two standard benchmarks: the aPascal Dataset (20 categories, 64 attributes) and the Animals with Attributes (AWA) Dataset (10 animal classes, 85 attributes).
As shown in the results for the aPascal dataset, the xLR-Rel+ model achieved an overall accuracy of 63.10%, significantly outperforming the baseline Logistic Regression (59.76%) and previous state-of-the-art attribute methods. On the AWA dataset, xLR-Rel+ performed best in 26 out of 45 pairwise animal classification tasks, surpassing other LHI methods like SVM+ and Rank Transfer. Statistical analysis using the Wilcoxon rank-sum test confirmed these improvements were significant (p-value < 5%).
Discussion and Conclusion
By treating attributes as hidden information rather than an intermediate layer, the model gains the semantic knowledge of attributes without the risk of error propagation. This "semantic regularization" ensures that images sharing many attributes are treated similarly in the classifier's latent space, even if their raw pixel data differs.
While the current formulation relies on linear mapping and requires high-quality manual annotations for training, it demonstrates a powerful way to structure classifiers. Future work may involve extending this framework to non-linear architectures like Convolutional Neural Networks (CNNs) or utilizing other forms of privileged information, such as natural language descriptions or 3D models.
Summary updated