Abstract
We present a structured inference approach in deep neural networks for multiple attribute prediction. In attribute prediction, a common approach is to learn independent classifiers on top of a good feature representation. However, such classifiers assume conditional independence on features and do not explicitly consider the dependency between attributes in the inference process. We propose to formulate attribute prediction in terms of marginal inference in the conditional random field. We model potential functions by deep neural networks and apply the sum-product algorithm to solve for the approximate marginal distribution in feed-forward networks. Our message passing layer implements sparse pairwise potentials by a softplus-linear function that is equivalent to a higher-order classifier, and learns all the model parameters by end-to-end back propagation. The experimental results using SUN attributes and CelebA datasets suggest that the structured inference improves the attribute prediction performance, and possibly uncovers the hidden relationship between attributes.
Introduction
In standard multi-label classification, most deep learning models utilize a "fork" architecture: a shared backbone extracts features, followed by independent sigmoid classifiers that predict each attribute separately. This approach relies on the fundamental assumption of conditional independence.
In reality, attributes are often highly correlated or mutually exclusive. For instance, a scene is unlikely to be both "indoor" and "outdoor" simultaneously, while attributes like "wearing lipstick" and "heavy makeup" frequently co-occur. Standard models rely entirely on internal feature representations to capture these relationships implicitly. This paper argues that explicitly modeling these inter-attribute dependencies during the inference process leads to more robust and accurate predictions.
Methodology
The authors propose a structured inference approach using Conditional Random Fields (CRFs) integrated directly into the neural network architecture. The framework consists of several key innovations:
Potential Functions via CNNs
Instead of using fixed weights for attribute relationships, the model uses the neural network to predict "potential functions" $\phi_c(\mathbf{x}_c|\mathbf{z})$. They employ a softplus-linear function ($\log(1 + \exp(\dots))$) to ensure these potentials remain positive, a mathematical requirement for valid probabilistic inference.
Unrolled Message Passing
To calculate the marginal probability of each attribute, the system utilizes the sum-product algorithm. Crucially, the iterations of this algorithm are "unrolled" into a fixed number of layers within the network. This transformation converts a complex optimization problem into a standard feed-forward computation, allowing the entire system to be trained end-to-end using backpropagation.
Sparse Factor Graphs
Modeling every possible pair of attributes is computationally expensive ($O(N^2)$). To maintain efficiency, the authors construct a sparse graph where each attribute is linked only to its $K$ most correlated neighbors, determined from training data statistics. This ensures the model remains scalable even as the number of attributes grows.
Experimental Results
The model was evaluated on two major benchmarks: the SUN Attribute Dataset (102 scene attributes) and the CelebA Dataset (40 facial attributes).
Performance Gains
On the SUN Attribute dataset, the "Linear CRF" model achieved an F1 score of 0.4418, a significant improvement over the baseline sigmoid model's 0.4105. The structured approach proved particularly effective at increasing recall; if visual evidence for one attribute was weak, the model could use the presence of a highly correlated neighbor to correctly infer the missing label.
Qualitative Success
As seen in Figure 3, the CRF model excels at "filling in the blanks." In one example, the baseline model failed to identify "man-made" in an indoor scene. The CRF model correctly inferred it by recognizing the scene was "enclosed" and featured "indoor lighting," leveraging the learned semantic relationships between these attributes.
Discussion and Future Work
The authors found that while increasing model capacity (e.g., moving from ResNet-18 to ResNet-50) improves results, the structured CRF layer provides a performance boost on top of that extra capacity. This suggests that the layer captures semantic logic that raw features alone do not.
However, limitations remain. Unrolling too many message-passing steps can lead to vanishing gradients, making a "width over depth" strategy (more pairwise relationships rather than more iterations) the current best practice. Future work may explore higher-order potentials (relationships between three or more attributes) or dynamic graphs that adapt based on the specific image being processed.
Summary updated