Abstract
Label distribution learning (LDL) is a newly arisen machine learning method that has been increasingly studied in recent years. In theory, LDL can be seen as a generalization of multi-label learning. Previous studies have shown that LDL is an effective approach to solve the label ambiguity problem. However, the dramatic increase in the number of possible label sets brings a challenge in performance to LDL. In this paper, we propose a novel label distribution learning algorithm to address the above issue. The key idea is to exploit correlations between different labels. We encode the label correlation into a distance to measure the similarity of any two labels. Moreover, we construct a distance-mapping function from the label set to the parameter matrix. Experimental results on eight real label distributed data sets demonstrate that the proposed algorithm performs remarkably better than both the state-of-the-art LDL methods and multi-label learning methods.
Introduction: Beyond Binary Classification
In traditional machine learning, classification tasks are typically viewed through a binary lens: an instance either belongs to a class or it does not. This is known as single-label or multi-label learning. However, real-world scenarios often involve "label ambiguity," where multiple labels apply to an instance with varying degrees of relevance.
Consider a facial expression: a person might be 70% happy, 20% surprised, and 10% fearful. Label Distribution Learning (LDL) addresses this by assigning a probability distribution across all possible labels rather than a simple binary set. While powerful, LDL faces a scalability challenge: as the number of labels increases, the output space grows exponentially.
The LDLLC Algorithm
The authors propose LDLLC (Label Distribution Learning by Exploiting Label Correlations) to mitigate the complexity of LDL by leveraging the inherent relationships between labels.
1. Objective Function and Correlation Encoding
Most LDL methods minimize the Kullback-Leibler (KL) divergence between the predicted and ground-truth distributions. LDLLC innovates by adding a correlation-based regularization term. By calculating Pearson’s correlation coefficients, the algorithm identifies labels that naturally cluster together. If two labels are highly correlated, the model encourages their corresponding parameters to be closer in the parameter space, effectively reducing the effective search space and improving generalization.
2. Maximum Entropy and Optimization
The model assumes a maximum entropy distribution for conditional probabilities. To find the optimal parameters, the researchers utilize L-BFGS, a quasi-Newton optimization algorithm well-suited for high-dimensional spaces.
Experimental Setup
The algorithm was evaluated on eight biological datasets derived from the yeast Saccharomyces cerevisiae. These datasets (e.g., Alpha, Cdc, Elu) contain gene expression levels across discrete time points, where the "description degree" of a label is the expression intensity at that specific time. Each dataset consists of 2,465 samples with 24 features, with the number of labels varying from 4 to 18.
Results and Discussion
LDLLC was benchmarked against seven state-of-the-art LDL algorithms and three Multi-Label Learning (MLL) methods.
Performance in LDL Tasks
As shown in the experimental results, LDLLC achieved superior performance across multiple metrics, including Euclidean distance and KL divergence. It secured the top rank on four datasets and remained highly competitive on the others.
Performance in MLL Tasks
To compare against MLL baselines, the authors used a "virtual label" ($y_0$) as a dynamic threshold. Labels with intensities exceeding $y_0$ were considered relevant. Under this framework, LDLLC consistently outperformed established MLL methods like ML-KNN and Rank-SVM, proving that the rich information in label distributions can enhance even binary classification tasks.
Conclusion
The LDLLC algorithm demonstrates that exploiting label correlations is a vital strategy for improving Label Distribution Learning. By integrating relational information into the learning process, the model achieves higher accuracy and better handles label ambiguity. Future work may explore non-linear correlation metrics to capture even more complex relationships between labels.
Reference: Jia, X., Li, W., Liu, J., & Zhang, Y. (2018). Label Distribution Learning by Exploiting Label Correlations. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18).
Summary updated