Abstract
Deep Neural Networks (DNNs) often struggle with distribution drift when applied to new domains. This paper presents LearnRisk-TC, a novel framework that treats transfer learning as a risk-minimization problem. By building an external risk model to identify potential mispredictions, the framework enables effective adaptation to target domains even with extremely limited labeled samples (under 500). Experimental results show that LearnRisk-TC consistently outperforms standard baselines and state-of-the-art models like BERT and RoBERTa in low-data scenarios.
Introduction
The brittleness of DNNs under distribution drift is a significant hurdle in real-world NLP. A sentiment classifier trained on one domain often fails on another due to shifts in vocabulary and context. Traditional transfer learning requires substantial labeled data for fine-tuning, which is often unavailable. LearnRisk-TC addresses this by focusing on "model risk"—the likelihood of a model making an error—rather than just mining source knowledge.
Methodology
LearnRisk-TC: Risk Analysis Component
Instead of relying on often over-confident internal probability scores, LearnRisk-TC builds a dedicated risk model through a three-step process:
- Risk Feature Generation: Extracts interpretable rules based on neighborhood consistency (KNN) and class centroids using both statistical features (TF-IDF) and DNN-based embeddings.
- Risk Model Construction: Utilizes concepts from financial "Value-at-Risk" (VaR) theory to aggregate features and estimate the maximum potential "loss" (misprediction probability).
- Risk Model Training: Employs a learn-to-rank objective to ensure that samples the model is likely to get wrong are ranked higher in risk than those it gets right.
Risk-Based Transfer Learning
The adaptation process occurs in two phases. Phase 1 involves standard training on the source dataset. Phase 2 fine-tunes the model on the target dataset by minimizing the misprediction risk identified by LearnRisk-TC, effectively teaching the model to avoid specific types of mistakes in the new environment.
Experimental Setup
The approach was evaluated on three benchmark news datasets: 20News, BBC, and AgNews. Six dataset pairs were created to simulate distribution drift. Crucially, validation data for the target domain was limited to fewer than 500 samples to test performance under extreme data scarcity.
Key Results
Superior Risk Detection
LearnRisk-TC demonstrated a high capability for identifying mispredictions. It achieved AUROC scores exceeding 0.9 on five out of six datasets, significantly outperforming the "TrustScore" baseline.
Transfer Learning Performance
In transfer learning benchmarks, LearnRisk-TC outperformed large-scale models including BERT, RoBERTa, and XLNet. For instance, on the 20News-BBC task, it achieved 93.22% accuracy, far surpassing other low-data methods like UST (70.93%). The model also proved robust to significant reductions in validation data, maintaining stable performance even with a 70% cut in supervision.
Discussion and Conclusion
LearnRisk-TC introduces a creative intersection of finance (VaR) and NLP to quantify model uncertainty. By focusing on interpretable risk features, it provides a more transparent path to domain adaptation than black-box methods. While currently focused on news classification, future work aims to extend this risk-based analysis to medical and legal domains, as well as to Large Language Models (LLMs).
Summary updated