Abstract
In recent years, there has been a great deal of progress in describing objects with attributes. Attributes have proven useful for object recognition, image search, face verification, image description, and zero-shot learning. Typically, attributes are either binary or relative: they describe either the presence or absence of a descriptive characteristic, or the relative magnitude of the characteristic when comparing two exemplars. However, prior work fails to model the actual way in which humans use these attributes in descriptive statements of images. Specifically, it does not address the important interactions between the binary and relative aspects of an attribute. In this work we propose a spoken attribute classifier which models a more natural way of using an attribute in a description. For each attribute we train a classifier which captures the specific way this attribute should be used. We show that as a result of using this model, we produce descriptions about images of people that are more natural and specific than past systems.
Introduction: The Binary vs. Relative Dichotomy
In computer vision, attributes are typically treated as either binary (e.g., "this person is wearing glasses") or relative (e.g., "this person is more smiling than that one"). However, human language is more nuanced. We do not use relative statements for everything; for example, one would rarely say someone is "more wearing glasses" than another person.
Conversely, humans often use relative statements only when a certain binary threshold is met. For instance, saying "Tom is smiling more than Susan" usually implies that both are smiling. If neither were smiling, a human would likely not bother with the comparison. Prior models failed to capture these interactions, often producing descriptions that felt technically true but socially awkward or "robotic."
The Spoken Attribute Framework
The authors propose a two-level classifier to predict the most natural "spoken attribute" statement for a pair of images, moving beyond the rigid dichotomy of traditional classifiers.
Methodology
- Semantic Feature Extraction: Traditional attribute classifiers are trained first. For each attribute, a linear SVM is used for binary classification and a RankSVM for relative ranking.
- The Spoken Classifier: Instead of raw pixels, the second stage takes the outputs (SVM scores) of the first stage as input. This 5-dimensional "semantic" vector includes binary scores for both images, individual ranking scores, and the difference between them.
- Multiclass Prediction: A multiclass SVM then predicts which of six natural language statements a human would be most likely to use:
- Both have the attribute.
- Neither has it.
- Person A has it, B does not.
- Person B has it, A does not.
- Person A has it more than B.
- Person B has it more than A.
Attribute "Personalities" and Insights
A key finding of this research is that different attributes follow distinct usage patterns. For example, "Glasses" is used almost exclusively in binary terms. In contrast, "Smiling" and "Teeth Visible" are frequently used in relative terms, but typically only when the attribute is present in both subjects.
Furthermore, the research highlights a flaw in traditional datasets: "forced choice" rankings. Traditional relative attribute datasets force workers to rank images even when a comparison is unnatural. The Spoken Attribute model avoids this by learning when a comparison is actually appropriate to verbalize.
Experimental Results
The researchers collected a new dataset of 600 face pairs and six facial attributes (bald, beard, glasses, male, smiling, teeth visible) to validate their approach.
- Predictive Accuracy: The two-stage "Spoken Attributes" classifier significantly outperformed baselines, achieving an average accuracy of 73% across all attributes in predicting the preferred human description.
- Image Search Enhancement: The model improves image search by "reading between the lines." When a user searches for a relative query like "A is more bearded than B," the model correctly prioritizes images where both people actually have beards, recognizing the implicit binary information contained within the relative statement.
- Natural Language Generation: Qualitatively, the model successfully avoided unnatural descriptions, such as applying relative comparisons to binary-heavy attributes like glasses.
Conclusion and Future Work
While the current model is limited to a fixed vocabulary of six attributes and six statement templates, it represents a significant step toward more human-like image description. Future work may address the "saliency" problem—determining which attribute is most important to mention in a given context—and expanding the framework to handle free-form natural language.
Summary updated