TL;DR
This paper introduces Deep Probabilistic Logic (DPL) and Self-Supervised Self-Supervision (S4) to tackle the data annotation bottleneck in deep learning. DPL unifies various forms of "weak" supervision (like distant supervision and data programming) with deep learning for task-specific self-supervised learning. S4 further enhances this by enabling the automatic discovery of new self-supervision signals, significantly reducing the need for human-labeled data.
What Problem Does This Paper Address?
Deep learning models excel at many tasks but require vast amounts of labeled data, which is expensive and time-consuming to acquire, especially in specialized domains like biomedicine. While self-supervised learning has emerged as a solution, existing methods often focus on learning general-purpose representations from unlabeled data (e.g., masked language modeling) and still require task-specific labeled data for fine-tuning. This paper addresses the challenge of leveraging domain expertise and readily available but noisy knowledge sources to create task-specific training data automatically, thereby alleviating the supervision bottleneck.
Methods
The core contribution is the Deep Probabilistic Logic (DPL) framework. DPL combines probabilistic logic with deep learning to enable task-specific self-supervision. It models label decisions as latent variables and uses probabilistic logic (specifically, Markov logic) to represent diverse self-supervision signals (e.g., labeling functions, joint inference constraints). DPL trains a deep neural network end-to-end using a variational Expectation-Maximization (EM) algorithm.
Building upon DPL, the authors introduce Self-Supervised Self-Supervision (S4). S4 extends DPL by enabling the automatic learning of new self-supervision signals. It iteratively proposes new self-supervision rules, which are either directly incorporated (structured self-training) or verified by a human expert (feature-based active learning). This allows S4 to discover and refine its own supervision, further reducing reliance on manual annotation.
Datasets
The paper evaluates its methods on several datasets:
- Biomedical Machine Reading: Specifically, cross-sentence relation extraction and entity linking tasks.
- Text Classification: IMDb, Stanford Sentiment Treebank (StanSent), and Yahoo! Answers datasets.
Key Insights and Ideas
- Task-Specific Self-Supervision is Powerful: Unlike general self-supervised learning, tailoring supervision to the specific task, even with noisy sources, can yield significant performance gains.
- Probabilistic Logic as a Unifying Framework: DPL elegantly combines diverse, potentially conflicting, weak supervision signals into a coherent probabilistic model.
- Automated Self-Supervision Discovery: S4's ability to learn new self-supervision rules is a crucial step towards truly autonomous learning, moving beyond relying solely on pre-defined human expertise.
- Structured Self-Training and Active Learning Synergy: Combining these two approaches within S4 allows for both efficient exploitation of existing signals and intelligent acquisition of new ones, optimizing human effort.
- Leveraging Domain Expertise Efficiently: The methods allow domain experts to contribute in more flexible ways (e.g., labeling functions, verifying proposed rules) rather than painstakingly labeling individual instances.
Key Results
- Biomedical Machine Reading: DPL substantially improved accuracy in cross-sentence relation extraction and entity linking, often matching or exceeding supervised methods with a fraction of human effort. For instance, DPL improved sample precision by ten absolute points and raised absolute recall by 25% compared to a state-of-the-art system.
- Text Classification: S4-SST (a variant of S4 focusing on structured self-training) achieved high accuracy on IMDb and Stanford datasets, often outperforming self-training with significantly more labeled data. For example, on IMDb, S4-SST attained 86% test accuracy with just six self-supervised tokens, outperforming self-training with 100 labeled examples by 16 absolute points.
- Efficiency: The methods demonstrated scalability, with training times around 3.5 hours for relation extraction and 2.5 hours for entity linking on PubMed-scale experiments using 25 CPU cores and one GPU.
Limitations and Future Work
- Computational Cost: While efficient, inference in probabilistic logic can still be computationally challenging, especially with high-order factors. The paper relies on approximations like loopy belief propagation.
- Noise Tolerance: While S4 is more noise-tolerant than DPL, the initial self-supervision quality still impacts performance.
- Future Directions: The authors suggest exploring:
- Combining DPL with deep generative models and probabilistic programming.
- Further investigation into combining structure learning and active learning in S4.
- Developing more sophisticated self-supervision classes and proposal algorithms.
- Applying these methods to other domains beyond NLP.
Summary updated