Abstract
Large-scale multi-label learning (LMLL) aims to annotate relevant labels from a large number of candidates for unseen data. Due to the high dimensionality in both feature and label spaces in LMLL, the storage overheads of LMLL models are often costly. This paper proposes a POP (joint label and feature Parameter OPtimization) method. It tries to filter out redundant model parameters to facilitate compact models. Our key insights are as follows. First, we investigate labels that have little impact on the commonly used LMLL performance metrics and only preserve a small number of dominant parameters for these labels. Second, for the remaining influential labels, we reduce spurious feature parameters that have little contribution to the generalization capability of models, and preserve parameters for only discriminative features. The overall problem is formulated as a constrained optimization problem pursuing minimal model size. In order to solve the resultant difficult optimization, we show that a relaxation of the optimization can be efficiently solved using binary search and greedy strategies. Experiments verify that the proposed method clearly reduces the model size compared to state-of-the-art LMLL approaches, in addition, achieves highly competitive performance.
Introduction
In Large-Scale Multi-Label Learning (LMLL), the objective is to tag data with the most relevant subset of labels from a massive pool—such as categorizing a Wikipedia article from thousands of possible tags. However, LMLL suffers from a "double curse of dimensionality": millions of input features and thousands of potential output labels. Traditional methods, like training separate binary classifiers for every label, lead to astronomical model sizes (often tens of gigabytes). This makes deployment on memory-constrained "edge devices" like smartphones or IoT sensors nearly impossible.
Previous compression attempts often relied on simplistic "low-rank" assumptions or treated all labels as equally important, which frequently degraded the accuracy of rare but critical labels. The POP (joint label and feature Parameter OPtimization) method addresses this "storage explosion" by selectively pruning model parameters based on their actual utility.
The POP Framework
The authors introduce POP, a two-stage optimization framework designed to create a "compact" version of a pre-trained model ($M$).
1. Label Parameter Optimization
Not all labels impact performance metrics equally. The authors mathematically prove that a label's impact on common LMLL metrics (like Propensity Scored Precision) is bounded by its frequency and its "propensity weight." Using this insight, they rank labels and use a binary search strategy to find the optimal threshold for pruning parameters from low-impact labels without violating a performance constraint ($\epsilon$).
2. Feature Parameter Optimization
For the influential labels that remain, POP identifies "spurious" features—those that add noise rather than discriminative power. They formulate this as an $L_0$-norm constrained optimization problem. Since $L_0$ is NP-hard to solve directly, they use a relaxation technique that zeros out feature weights falling within a specific range $[-\sqrt{\lambda}, \sqrt{\lambda}]$.
By combining these stages, POP performs a "surgical" pruning that keeps only the most vital connections between specific features and specific labels.
Key Results and Insights
The researchers evaluated POP on four standard benchmarks: Bibtex, Delicious, Eurlex, and Wiki10.
- Massive Compression: On the
wiki10dataset, POP reduced the model size from 23.50 GB to just 67.50 MB—a reduction of over 99%—while maintaining nearly identical precision. - Metric-Aware Pruning: Unlike standard compression algorithms that look only at weight magnitudes, POP evaluates the utility of a parameter relative to the final evaluation metric. If a label doesn't significantly move the "Propensity Scored Precision" score, its parameters are prime candidates for aggressive pruning.
- Superiority over SOTA: Compared to other compressed baselines like
PD-SparseandDiSMEC, POP consistently achieved a better balance between model size and predictive accuracy. For instance, oneurlex, POP was roughly 4x smaller thanDiSMECwhile maintaining higher precision.
Conclusion
POP demonstrates that structural assumptions like "low-rank" are not necessary for effective model compression in LMLL. By focusing on the utility of parameters relative to performance metrics, POP achieves state-of-the-art compression ratios. While the method is sensitive to the feature sparsity parameter ($\lambda$) and depends on the quality of the initial pre-trained model, it represents a significant step toward deploying large-scale models on edge devices.
Summary updated