TL;DR
This paper tackles the challenge of transferring a manually developed coding system for analyzing literary translation trends from one Hungarian literary journal to another. The authors demonstrate that BERT models, with extensive hyperparameter tuning and specialized loss functions, can successfully adapt this coding system to a new domain, even with domain shift and imbalanced data. This work is significant for its practical application in computational social science and its methodological contributions to domain adaptation for NLP tasks.
What Problem Does This Paper Address?
Analyzing cultural and social trends often requires nuanced textual analysis. In this case, the researchers aimed to track perceptions of literary translation in Hungary around the 1989 political transformation. To do this, they developed a detailed coding system for paragraphs in a literary journal. However, applying this system to a different, albeit related, literary journal (the "target domain") presents a significant challenge: domain shift. This means the language, style, and specific topics might differ, potentially hindering the performance of models trained on the original corpus. Furthermore, the coding system itself is highly imbalanced, with some labels being far more frequent than others, which can bias machine learning models. The paper addresses how to effectively transfer this complex, imbalanced coding system to a new domain using BERT models, while also ensuring the reliability of the evaluation process.
Methods
The core of this research involves training and evaluating BERT models for text classification. Key methodological aspects include:
- Domain Adaptation: The researchers first adapt a pre-trained BERT model (PULI-BERT-Large) to the combined corpus of the source and target journals. This "pretraining" step helps the model learn the general linguistic characteristics of the domain.
- Finetuning with Specialized Techniques: For the actual classification task, the models are finetuned using:
- 10-fold Crossvalidation: This standard technique provides robust evaluation by splitting the training data into ten parts, training on nine, and testing on the tenth, repeating ten times. It also allows for ensemble creation.
- Population-Based Training (PBT): Used for hyperparameter optimization, PBT trains many models in parallel and adapts hyperparameters on the fly, making it scalable and efficient.
- Imbalanced Loss Functions: For the highly imbalanced content labels, Focal Loss is used to down-weight easy examples and focus training on hard, misclassified ones. For context labels, a combination of Label Distribution-Aware Margin Loss (LDAM) and a Confident output distribution Penalty (CP) is employed to handle imbalance and improve confidence.
- Domain Shift Evaluation: To specifically test the models' ability to generalize, a separate test set is sampled from the target domain. A novel method is introduced to estimate confidence intervals for evaluation metrics based on sample size, helping to determine an adequate test set size for manual validation.
- Importance Sampling: Since the target domain lacks labels, importance sampling is used to draw paragraphs for the test set, prioritizing those with less frequent predicted labels to better represent the imbalanced nature of the task.
Datasets
- Source Domain Corpus: Paragraphs from the Hungarian literary journal Alföld (issues from 1980-1999), manually annotated with a coding system.
- Target Domain Corpus: Paragraphs from the Hungarian literary journal Nagyvilág (issues from 1980-1999), used for evaluating domain adaptation and transfer learning.
- OCR Corpus: A corpus of OCR-ed journals from the same period but with different subject matter, used for domain adaptation experiments to assess the impact of OCR quality and domain shift.
Key Insights and Ideas
- BERT's Adaptability: The study strongly suggests that BERT models, with proper tuning, can effectively transfer complex coding systems across domains, even when there's a significant "domain shift."
- Imbalance is a Real Challenge, But Solvable: The use of specialized loss functions like Focal Loss and LDAM+CP is crucial for achieving good performance on highly imbalanced datasets, preventing models from simply predicting the majority class.
- Domain Adaptation Matters: Adapting BERT models to the target domain's text (even if it's OCR-ed text from a different subject) provides a substantial performance boost compared to using off-the-shelf models.
- Ensembling for Robustness: Combining multiple models trained via 10-fold crossvalidation (ensembling) leads to more robust evaluation scores and better overall predictions.
- Context vs. Content Labels: The paper highlights that context labels proved more challenging to transfer than content labels, suggesting differences in how these label types capture information and their susceptibility to domain shift.
Key Results
- Content Labels: The BERT ensemble achieved a ROC AUC of 0.8757 ± 0.0252 on the content labels in the target domain, demonstrating successful transfer.
- Context Labels: The performance on context labels was significantly lower, with a balanced accuracy of 0.3287 ± 0.1297. This led the authors to focus their findings primarily on the content labels.
- Domain Adaptation Gains: Domain adaptation to the target corpus reduced the perplexity of the PULI-BERT-Large model from 43.07 to 2.88. Adaptation to OCR-ed text also showed significant improvements over no adaptation.
- Baseline Comparison: BERT models significantly outperformed traditional low-cost baseline methods (TF-IDF, fastText embeddings with gradient boosting).
Limitations and Future Work
- Context Label Performance: The significantly lower performance on context labels is a key limitation. The authors hypothesize potential reasons include data scarcity, inadequate context window size, and novel patterns in the target domain that differ from the training data. Further investigation is needed here.
- Manual Annotation Effort: Manual validation for evaluating domain shift is resource-intensive. The proposed method for estimating confidence intervals based on sample size helps optimize this process, but it remains a bottleneck.
- Llama 2 Performance: Preliminary experiments with Llama 2 models showed that BERT models still surpassed them in performance for this task, though the authors acknowledge that instruction finetuning might yield better results for generative models.
- Future Work: The authors suggest further investigation into the reasons for misclassification on context labels, exploring more advanced domain adaptation techniques, and potentially using generative models with instruction finetuning for tasks requiring deeper linguistic and factual knowledge. They also note that the coding system itself is still evolving, with potential for further refinement and more annotators in future stages.
Summary updated