Introduction
Deep learning models typically require vast amounts of high-quality labeled data, which is often prohibitively expensive to acquire. Weak supervision has emerged as a popular alternative, where "labeling functions" (LFs)—such as keyword searches, pattern matching, or external knowledge bases—are used to automatically annotate large unlabeled datasets. However, these models often suffer from a "shortcut" problem: they over-rely on the specific, noisy signals of the LFs rather than learning the deeper semantic meaning of the task.
KnowMAN (Knowledge-based Weakly Supervised Multinomial Adversarial Networks) is a framework designed to prevent this overfitting. By employing an adversarial training strategy, KnowMAN forces models to ignore biases inherent in specific labeling heuristics, resulting in more robust and generalizable feature representations.
The Problem: Overfitting to Heuristics
When a model is trained on weakly supervised data, it often becomes "lazy." For instance, if an LF labels a movie review as "positive" based on the word "excellent," the model might learn that the presence of that specific word is the only indicator of sentiment. This prevents the model from generalizing to reviews that express positivity through different vocabulary or complex sentence structures.
Previous solutions have attempted to model LF noise (like Snorkel) or physically remove keywords from the text ("hard deletion"). However, hard deletion can strip away useful context. KnowMAN proposes a "soft deletion" approach, using adversarial training to blur LF-specific signals in the latent space without removing the raw information from the input.
The KnowMAN Framework
KnowMAN treats labeling functions as distinct "domains" and applies an adversarial strategy inspired by domain adaptation. The architecture consists of three primary modules:
- Shared Feature Extractor ($F_s$): Processes input text (via TF-IDF or transformers like DistilBERT) into a vector representation.
- Task Classifier ($C$): Predicts the target label (e.g., Spam vs. Not Spam).
- LF-Discriminator ($D$): An adversarial component that attempts to identify which specific labeling function generated the label for a given input.
During training, KnowMAN utilizes gradient reversal. While the classifier $C$ works to minimize task error, the feature extractor $F_s$ is simultaneously trained to maximize the error of the LF-discriminator $D$. This adversarial pressure forces $F_s$ to discard features unique to a specific LF (like a specific keyword) and instead focus on signals that are generally associated with the output label across all LFs. A hyperparameter, $\lambda$, controls the strength of this adversarial blurring.
Experimental Results
The authors evaluated KnowMAN against standard benchmarks, including YouTube Spam detection, news-based Spouse relation extraction, and IMDb sentiment analysis. The IMDb dataset served as a particular "stress test," featuring over 6,700 LFs.
KnowMAN consistently outperformed baselines such as Snorkel and fine-tuned DistilBERT by margins of 2% to 30%:
- Spam: Achieved 0.94 accuracy using TF-IDF, significantly higher than the 0.87 baseline.
- Spouse: Reached an F1-score of 0.39, nearly doubling the performance of standard weak supervision (0.20).
- IMDb: Achieved 0.77 accuracy. Notably, Snorkel failed to run on this dataset due to memory constraints (exceeding 32GB RAM) caused by the high number of LFs, whereas KnowMAN’s neural architecture scaled effortlessly.
Key Insights and Limitations
One of the most significant findings is that treating LFs as domains allows the model to learn "LF-invariant" features. This latent-space approach preserves more information than physical masking while preventing the model from becoming dependent on specific shortcuts.
However, the framework does have limitations. The optimal value for the adversarial hyperparameter $\lambda$ varies by dataset, requiring a validation set for tuning—a resource not always available in zero-label scenarios. Additionally, performance in transformer-based versions was sometimes limited by the need to truncate long text sequences to fit GPU memory, occasionally allowing simpler TF-IDF versions to perform better on long-form content like movie reviews.
Future work may involve integrating "mixture of experts" modules to capture specialized domain features that a single shared extractor might overlook.
Summary updated