Abstract
In computer vision, image datasets used for classification are naturally associated with multiple labels and comprised of multiple views, because each image may contain several objects (e.g. pedestrian, bicycle and tree) and is properly characterized by multiple visual features (e.g. color, texture and shape). Currently available tools ignore either the label relationship or the view complementary. Motivated by the success of the vector-valued function that constructs matrix-valued kernels to explore the multi-label structure in the output space, we introduce multi-view vector-valued manifold regularization (MV³MR) to integrate multiple features. MV³MR exploits the complementary property of different features and discovers the intrinsic local geometry of the compact support shared by different features under the theme of manifold regularization. We conducted extensive experiments on two challenging, but popular datasets, PASCAL VOC' 07 (VOC) and MIR Flickr (MIR), and validated the effectiveness of the proposed MV³MR for image classification.
Introduction
Modern image classification tasks face two primary challenges: multi-label complexity and multi-view diversity. A single image often contains multiple semantic concepts (e.g., "sea" and "sky"), and no single feature descriptor—such as SIFT for local texture or GIST for global layout—can capture all relevant information. Traditional approaches often treat labels as independent or simply concatenate features into a high-dimensional vector, which can lead to overfitting and neglect the inherent structure of the data. This paper introduces Multi-view Vector-valued Manifold Regularization (MV³MR) to address these issues by learning label correlations and intelligently integrating diverse feature views.
The MV³MR Framework
The MV³MR framework integrates several advanced machine learning techniques to improve classification performance:
- Vector-valued Functions: By predicting a vector of labels simultaneously, the model utilizes matrix-valued kernels to explicitly encode correlations between different labels.
- Manifold Regularization: This component leverages the geometric structure of both labeled and unlabeled data. It assumes that images close to each other in the feature space should share similar labels, allowing the model to perform well even with limited labeled training data.
- Multi-Kernel Learning (MKL): Instead of feature concatenation, MV³MR assigns individual kernels to each view. The algorithm learns optimal weights for these kernels and their corresponding graph Laplacians, automatically identifying the most discriminative features for a specific task.
- MV³LSVM: The authors derive a Support Vector Machine formulation using hinge loss, providing a robust classification framework.
Experimental Evaluation
The proposed framework was tested on the PASCAL VOC 2007 and MIR Flickr datasets using seven distinct feature views, including SIFT, GIST, and various color histograms (RGB, HSV, LAB). A key insight from the experiments is the importance of feature complementarity; for instance, while color features might fail to distinguish between similar scenes, global layout features like GIST often provide the necessary discriminative power.
Key Results
The experimental results demonstrate that the proposed MV³LSVM consistently outperforms state-of-the-art methods like SimpleMKL and multi-label compressed sensing (MLCS).
As shown in the figure above, MV³LSVM achieves significant improvements in mean Average Precision (mAP), especially when the number of labeled samples is small. On the PASCAL VOC dataset with only 100 labeled samples, the model showed an 8.1% improvement over the popular SimpleMKL algorithm. Furthermore, the model demonstrated superior ranking performance and strong generalization to out-of-sample data, proving its robustness in real-world scenarios.
Conclusion
MV³MR provides a powerful framework for multi-label, multi-view image classification by exploiting label correlations and the intrinsic geometry of data manifolds. While the performance gap narrows as labeled data becomes abundant, the framework's ability to learn from sparse labeled data and its robustness to initialization make it a valuable tool for modern computer vision tasks. Future work may extend this multi-view approach to other learning algorithms and explore optimizations for extremely large-scale datasets.
Summary updated