Knodle: Modular Weakly Supervised Learning with PyTorch

Anastasiia Sedova, Andreas Stephan, Marina Speranskaya, Benjamin Roth

TL;DR

Knodle is an open-source Python framework built on PyTorch that modularizes weak supervision by cleanly separating data abstractions, denoising methods, and deep learning prediction models. By providing a unified tensor-driven interface, it allows researchers to seamlessly swap and benchmark different denoising algorithms across diverse tasks. Along with the framework, the authors introduce WSCrossWeigh, a novel denoising algorithm that outperforms standard baselines on complex, large-scale datasets.

What Problem Does This Paper Address?

Deep learning's appetite for massive labeled datasets is a major bottleneck, as manual annotation is expensive and time-consuming. Weak supervision offers a workaround by automatically generating labels using heuristic rules, keyword lists, or external databases. However, these programmatic labels are inherently noisy.

While many denoising algorithms exist to clean up this data, they are typically tightly coupled to specific tasks or model architectures. For instance, some methods only look at correlations between heuristic rules, ignoring the underlying deep learning model entirely, while others are hardcoded into specific neural network layers. This lack of standardization makes it incredibly difficult for researchers to compare different denoising strategies fairly or transfer a successful method from one task to another.

Methods

Knodle solves this coupling problem by introducing a modular architecture that treats weak data annotations, deep learning models, and denoising methods as independent components.

Figure 1: The figure gives an overview of our system. (a) represents the preprocessed input, given as tensors.(b) resembles the internals of Knodle. The Trainer classes introduced in Section 3.2 handle transformation, denoising and model training. Note that these three steps could be performed subsequently or subsumed in a single training step. Then, (c) shows the output, a trained PyTorch model.

The framework relies on a simple, tensor-driven data abstraction. Instead of passing a single noisy label matrix, Knodle tracks fine-grained information using two binary matrices:

  1. $Z$ (samples $\times$ rules): Tracks which heuristic rules matched each data instance.
  2. $T$ (rules $\times$ classes): Maps which rules are indicative of which output classes.

By keeping these matrices separate, denoising algorithms can leverage the exact interplay between specific rules and the deep learning model's predictions.

Knodle includes several built-in Trainer classes (e.g., Majority Voting, $k$-NN Denoising, and Snorkel). Additionally, the authors introduce a new algorithm called WSCrossWeigh (Weak Supervision CrossWeigh). Inspired by crowdsourcing error-detection methods, WSCrossWeigh uses a rule-based cross-validation loop. It splits labeling rules into folds, trains a model on a subset of rules, and tests it on the remaining rule-matched samples. If the model's prediction contradicts the weak label, the sample is flagged as potentially mislabeled and its weight is reduced for the final training phase.

Datasets

The authors integrated four existing datasets into Knodle’s tensor format to serve as a benchmarking suite:

  • Spam: A small YouTube comments dataset for binary spam classification, highly sensitive to individual labeling errors.
  • Spouse: A highly skewed relation extraction dataset where over 90% of the test set represents a negative ("no-spouse") relation.
  • IMDb: The classic movie review sentiment dataset, characterized by a very high density of rule hits per sentence.
  • TAC-based RE: A massive, complex relation extraction dataset with 41 classes and over 180,000 rules, which includes an equal proportion of negative instances.
Summary of data statistics for the four benchmark datasets integrated into Knodle.

Key Insights and Ideas

  • No "One Size Fits All" Denoising: The experiments reveal that a denoising method's effectiveness depends heavily on dataset characteristics like rule density and class skewness.
  • The Danger of Over-Propagation: Geometric denoising methods like $k$-NN work well when weak labels are reasonably reliable. However, on datasets with dense rule hits (like IMDb), $k$-NN can propagate imprecise labels to neighbors, causing performance to plummet to the level of a random guess (50% accuracy).
  • Down-weighting vs. Changing Rules: WSCrossWeigh proved highly robust because it doesn't try to alter the rules themselves; instead, it down-weights the samples it lacks confidence in. This makes it far more resilient against extremely noisy or overgeneralized heuristics.

Key Results

The authors evaluated the denoising methods using a DistilBert model as the core predictor.

Quantitative evaluation results across different denoising modes and datasets.
  • Spam: Both $k$-NN and WSCrossWeigh achieved the highest accuracy at 0.94, significantly outperforming the simple majority vote baseline (0.81).
  • IMDb: WSCrossWeigh handled the dense rule noise exceptionally well, reaching 0.73 accuracy, whereas both $k$-NN and Snorkel stalled at 0.50.
  • TAC-based RE: On this massive dataset, WSCrossWeigh achieved the top F1-score of 0.26. Notably, Snorkel failed to run entirely due to the computational overhead of processing more than 180,000 rules without prior data manipulation.

Limitations and Future Work

While Knodle succeeds as a modular testbed, scalability remains a challenge for certain algorithms. For instance, standard $k$-NN required an approximate nearest neighbors workaround (Annoy) to handle the millions of instances in the TAC-based RE dataset, resulting in a drop in performance due to approximation losses.

Additionally, handling negative (unlabeled) instances still requires careful, task-specific configuration by the user. The authors hope the community will use Knodle's open-source repository to develop and contribute new denoising algorithms that can better navigate these large-scale computational trade-offs.

Summary updated

Abstract

Strategies for improving the training and prediction quality of weakly supervised machine learning models vary in how much they are tailored to a specific task or integrated with a specific model architecture. In this work, we introduce Knodle, a software framework that treats weak data annotations, deep learning models, and methods for improving weakly supervised training as separate, modular components. This modularization gives the training process access to fine-grained information such as data set characteristics, matches of heuristic rules, or elements of the deep learning model ultimately used for prediction. Hence, our framework can encompass a wide range of training methods for improving weak supervision, ranging from methods that only look at correlations of rules and output classes (independently of the machine learning model trained with the resulting labels), to those that harness the interplay of neural networks and weakly labeled data. We illustrate the benchmarking potential of the framework with a performance comparison of several reference implementations on a selection of datasets that are already available in Knodle. The framework is published as an open-source Python package knodle and available at https://github.com/knodle/knodle.
BibTeX
@inproceedings{sedova2021knodle,
  author = {Anastasiia Sedova and Andreas Stephan and Marina Speranskaya and Benjamin Roth},
  title = {{Knodle: Modular Weakly Supervised Learning with PyTorch}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure