Combining Probabilistic Logic and Deep Learning for Self-Supervised Learning

Hoifung Poon, Hai Wang, Hunter LangarXiv 2021 · OpenReview

TL;DR

This paper introduces Deep Probabilistic Logic (DPL) and Self-Supervised Self-Supervision (S4) to tackle the data annotation bottleneck in deep learning. DPL unifies various forms of "weak" supervision (like distant supervision and data programming) with deep learning for task-specific self-supervised learning. S4 further enhances this by enabling the automatic discovery of new self-supervision signals, significantly reducing the need for human-labeled data.

What Problem Does This Paper Address?

Deep learning models excel at many tasks but require vast amounts of labeled data, which is expensive and time-consuming to acquire, especially in specialized domains like biomedicine. While self-supervised learning has emerged as a solution, existing methods often focus on learning general-purpose representations from unlabeled data (e.g., masked language modeling) and still require task-specific labeled data for fine-tuning. This paper addresses the challenge of leveraging domain expertise and readily available but noisy knowledge sources to create task-specific training data automatically, thereby alleviating the supervision bottleneck.

Methods

The core contribution is the Deep Probabilistic Logic (DPL) framework. DPL combines probabilistic logic with deep learning to enable task-specific self-supervision. It models label decisions as latent variables and uses probabilistic logic (specifically, Markov logic) to represent diverse self-supervision signals (e.g., labeling functions, joint inference constraints). DPL trains a deep neural network end-to-end using a variational Expectation-Maximization (EM) algorithm.

Building upon DPL, the authors introduce Self-Supervised Self-Supervision (S4). S4 extends DPL by enabling the automatic learning of new self-supervision signals. It iteratively proposes new self-supervision rules, which are either directly incorporated (structured self-training) or verified by a human expert (feature-based active learning). This allows S4 to discover and refine its own supervision, further reducing reliance on manual annotation.

Datasets

The paper evaluates its methods on several datasets:

  • Biomedical Machine Reading: Specifically, cross-sentence relation extraction and entity linking tasks.
  • Text Classification: IMDb, Stanford Sentiment Treebank (StanSent), and Yahoo! Answers datasets.

Key Insights and Ideas

  • Task-Specific Self-Supervision is Powerful: Unlike general self-supervised learning, tailoring supervision to the specific task, even with noisy sources, can yield significant performance gains.
  • Probabilistic Logic as a Unifying Framework: DPL elegantly combines diverse, potentially conflicting, weak supervision signals into a coherent probabilistic model.
  • Automated Self-Supervision Discovery: S4's ability to learn new self-supervision rules is a crucial step towards truly autonomous learning, moving beyond relying solely on pre-defined human expertise.
  • Structured Self-Training and Active Learning Synergy: Combining these two approaches within S4 allows for both efficient exploitation of existing signals and intelligent acquisition of new ones, optimizing human effort.
  • Leveraging Domain Expertise Efficiently: The methods allow domain experts to contribute in more flexible ways (e.g., labeling functions, verifying proposed rules) rather than painstakingly labeling individual instances.

Key Results

  • Biomedical Machine Reading: DPL substantially improved accuracy in cross-sentence relation extraction and entity linking, often matching or exceeding supervised methods with a fraction of human effort. For instance, DPL improved sample precision by ten absolute points and raised absolute recall by 25% compared to a state-of-the-art system.
  • Text Classification: S4-SST (a variant of S4 focusing on structured self-training) achieved high accuracy on IMDb and Stanford datasets, often outperforming self-training with significantly more labeled data. For example, on IMDb, S4-SST attained 86% test accuracy with just six self-supervised tokens, outperforming self-training with 100 labeled examples by 16 absolute points.
  • Efficiency: The methods demonstrated scalability, with training times around 3.5 hours for relation extraction and 2.5 hours for entity linking on PubMed-scale experiments using 25 CPU cores and one GPU.

Limitations and Future Work

  • Computational Cost: While efficient, inference in probabilistic logic can still be computationally challenging, especially with high-order factors. The paper relies on approximations like loopy belief propagation.
  • Noise Tolerance: While S4 is more noise-tolerant than DPL, the initial self-supervision quality still impacts performance.
  • Future Directions: The authors suggest exploring:
  • Combining DPL with deep generative models and probabilistic programming.
  • Further investigation into combining structure learning and active learning in S4.
  • Developing more sophisticated self-supervision classes and proposal algorithms.
  • Applying these methods to other domains beyond NLP.

Summary updated

Abstract

Deep learning has proven effective for various application tasks, but its applicability is limited by the reliance on annotated examples. Self-supervised learning has emerged as a promising direction to alleviate the supervision bottleneck, but existing work focuses on leveraging co-occurrences in unlabeled data for task-agnostic representation learning, as exemplified by masked language model pretraining. In this chapter, we explore task-specific self-supervision, which leverages domain knowledge to automatically annotate noisy training examples for end applications, either by introducing labeling functions for annotating individual instances, or by imposing constraints over interdependent label decisions. We first present deep probabilistic logic(DPL), which offers a unifying framework for task-specific self-supervision by composing probabilistic logic with deep learning. DPL represents unknown labels as latent variables and incorporates diverse self-supervision using probabilistic logic to train a deep neural network end-to-end using variational EM. Next, we present self-supervised self-supervision(S4), which adds to DPL the capability to learn new self-supervision automatically. Starting from an initial seed self-supervision, S4 iteratively uses the deep neural network to propose new self supervision. These are either added directly (a form of structured self-training) or verified by a human expert (as in feature-based active learning). Experiments on real-world applications such as biomedical machine reading and various text classification tasks show that task-specific self-supervision can effectively leverage domain expertise and often match the accuracy of supervised methods with a tiny fraction of human effort.
BibTeX
@misc{poon2021combining,
  author = {Hoifung Poon and Hai Wang and Hunter Lang},
  title = {{Combining Probabilistic Logic and Deep Learning for Self-Supervised Learning}},
  year = {2021},
  eprint = {2107.12591},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.12591},
}
Figure