TL;DR
This paper tackles the challenging task of automatically assigning subject tags to library records, especially in a multilingual setting with a massive number of potential subjects. The authors successfully applied contrastive learning with multilingual Sentence-BERT models, achieving significant improvements in recall and demonstrating the effectiveness of their approach for low-resource scenarios.
What Problem Does This Paper Address?
Manually tagging library bibliographic records with subject headings is a labor-intensive and time-consuming process. The SemEval-2025 Task 5, specifically sub-task 2, aims to automate this by aligning library records (TIBKAT collection) with a large, hierarchical subject vocabulary (GND Subjects). This task presents several significant challenges:
- Cross-disciplinary Document Coverage: The subject matter spans a wide range of disciplines, making it difficult for models to generalize.
- Multilingual Requirements: The data is available in both English and German, necessitating models that can handle multiple languages.
- Extreme Classification: The sheer scale of the GND Subject vocabulary, with over 200,000 subjects, makes traditional classification methods computationally expensive and prone to sparsity issues.
- Label Underutilization and Sparsity: A significant portion of the available subject labels are not present in the training data, and many subjects are associated with very few documents, exacerbating the sparsity problem.
Methods
The core of the YNU-HPCC team's approach is contrastive learning applied to multilingual Sentence-BERT (SBERT) models. Here's a breakdown of their methodology:
- Dual-Encoder Architecture: They use a Siamese-like architecture where two SBERT models (with shared parameters) encode the TIBKAT record and the GND Subject independently. The similarity between their embeddings is then used to determine the match.
- Semantic Encoding: Both the title and abstract of a TIBKAT record are concatenated and fed into the SBERT model. For GND Subjects, their name and any alternate names are similarly encoded.
- Contrastive Loss Function: They optimize a temperature-scaled contrastive loss function. This loss encourages similar pairs (a record and its correct subject) to have high similarity scores and dissimilar pairs to have low similarity scores.
- Training Strategies: Two distinct sampling strategies were explored to create training pairs:
- Aggregate Multi-label Sampling: All true subjects for a given record are aggregated into a single positive sample, and a random selection of incorrect subjects forms negative samples.
- Instance-wise Disaggregated Sampling: Each true subject for a record forms a separate positive sample, paired with a randomly selected negative subject.
- Multilingual SBERT Variants: They experimented with several multilingual SBERT models from the
sentence-transformerslibrary, including MiniLM and mpnet variants.
Datasets
The primary datasets used in this work are:
- TIBKAT Collection: This collection contains library bibliographic records that need to be tagged.
- GND Subjects: A large, hierarchical vocabulary of over 200,000 subjects used for tagging.
The task provides two datasets for evaluation:
- Core Dataset: Contains 79,427 subject labels.
- All Dataset: Contains 204,739 subject labels.
The authors note that due to an operational oversight, their best-performing model (PM-mpnet-v2) was not included in the final submission.
Key Insights and Ideas
- Contrastive Learning is Key for Low-Resource Multilingual Tasks: The paper strongly suggests that contrastive learning is a powerful technique for aligning semantic representations in multilingual, low-resource scenarios, which is crucial for tasks like subject tagging.
- Architecture Matters for Training Strategy: A surprising finding was that different SBERT architectures respond differently to training strategies. MiniLM-series models benefited more from multi-label sampling, while mpnet variants excelled with single-label sampling. This highlights the need for architecture-aware hyperparameter tuning.
- Zero-Shot Performance is a Good Indicator: The initial zero-shot performance of pre-trained models (as shown in Table 1) was a good predictor of their potential for this task, with MiniLM-series models showing particular promise.
- Top-k Thresholds Impact Metrics: The choice of 'k' (the number of top predicted subjects) significantly affects precision and recall. Precision decreases as 'k' increases, while recall increases. This implies that the optimal 'k' is application-dependent and requires balancing these trade-offs.
Key Results
- Significant Recall Improvement: The best-performing fine-tuned model (PM-MiniLM-L12-v2) achieved a 28.6% improvement in average recall on the core-test set, reaching 0.2252. On the all-test set, the recall was 0.1677.
- Model-Specific Gains:
- MiniLM-series models saw a +33.5% zero-shot recall improvement with multi-label training.
- mpnet variants achieved a +230.3% zero-shot recall improvement with single-label training.
- Baseline Performance: The paraphrase-multilingual-MiniLM-L12-v2 (PM-MiniLM-L12-v2) model demonstrated superior zero-shot recall (17.51%) compared to other candidates.
Limitations and Future Work
- Exclusion of Best Model: The authors acknowledge that their potentially superior PM-mpnet-v2 model was inadvertently excluded from the final submission due to an operational oversight.
- Domain Adaptation Challenges: The performance on the all-test dataset was comparatively lower than on the core-test dataset, suggesting potential domain adaptation challenges when moving from a more constrained set to a larger, more diverse one.
- Future Research Directions:
- Bidirectional Translation for Data Augmentation: Employing bidirectional translation to augment data could help address the issue of insufficient samples for certain subjects.
- Two-Stage Inference with LLMs: For inference, a two-stage approach combining SBERT for initial candidate selection (top 500 subjects) followed by a larger language model (LLM) for final selection (top 50 subjects) is proposed.
- Improved Loss Functions: Further research into optimizing the loss function and training set could address issues like similarity determination and hubness problems.
Summary updated