Abstract
Multi-label text classification often suffers from extreme data imbalance, where "head" categories are overrepresented and "tail" categories lack sufficient training data. We present the Hybrid-Siamese Convolutional Neural Network (HSCNN), an architecture that integrates standard discriminative learning for head categories with relational Siamese learning for tail categories. By employing a multi-task learning framework and a category-specific similarity measure, HSCNN significantly improves performance on underrepresented classes without compromising accuracy on frequent ones. Experimental results on RCV1 and Delicious datasets demonstrate substantial gains in Macro-F1 scores, particularly for tail categories.
Introduction
In real-world multi-label text classification, datasets typically follow a "long-tail" distribution. While standard models like Convolutional Neural Networks (CNNs) excel at learning features for data-rich head categories, they often fail to generalize to tail categories that may have only a few examples. Conversely, few-shot learning techniques like Siamese networks are effective for low-data regimes but can be inefficient for head categories. HSCNN addresses this disparity by proposing a hybrid architecture that leverages the complementary strengths of both approaches.
Methodology
The HSCNN architecture is designed around three primary technical innovations:
- Multi-task Architecture: The model simultaneously trains a standard CNN for classification and a Siamese structure for similarity detection. This joint training acts as a regularizer, ensuring the Siamese component benefits from general features learned from the head categories and preventing overfitting on small samples.
- Category-Specific Similarity: Unlike traditional Siamese networks that output a global similarity score, HSCNN introduces a category vector. This allows the model to determine if two instances are similar specifically regarding a particular category $C$, capturing the nuanced overlaps common in multi-label documents.
- Asymmetric Sampling: To effectively train the category-specific structure, the authors utilize a specialized sampling method maintaining a 1:2 ratio of similar to dissimilar triplets.
During inference, a threshold $N_\phi$ based on training instance counts determines the prediction strategy: head categories utilize the standard CNN output, while tail categories rely on the Siamese similarity output.
Experimental Setup
The model was evaluated on two benchmark datasets exhibiting extreme imbalance:
- RCV1: 103 categories of Reuters newswire articles, with frequencies ranging from over 10,000 to just one instance.
- Delicious: 983 categories of web bookmarks.
Results and Discussion
HSCNN consistently outperforms standard CNNs and baseline hybrid models across various metrics.
- Tail Category Performance: On the RCV1 dataset, the Macro-F1 score for tail categories jumped from 3.96% (standard CNN) to 12.80% with HSCNN.
- Overall Gains: The hybrid approach raised the overall Macro-F1 on RCV1 from 36.19% to 39.20%.
- Impact of Hybrid Threshold: The choice of the hybrid threshold $N_\phi$ is critical. As shown below, the model consistently outperforms standard single-task baselines across a range of thresholds and different loss functions.
Furthermore, the improvements were consistent regardless of the underlying loss function—Binary Cross-Entropy (BCE), Weighted Cross-Entropy (WCE), or Focal Loss (FL)—indicating that HSCNN provides a structural advantage that complements mathematical loss-tuning.
Conclusion
HSCNN successfully bridges the gap between discriminative and relational learning for imbalanced text classification. While the model requires manual threshold tuning and increased computational resources for triplet sampling, its ability to capture contextual similarity significantly boosts performance on the long tail. Future research may explore replacing the CNN backbone with Transformer-based architectures like BERT to further enhance feature extraction and generalization.
Summary updated