I Know Therefore I Score: Label-Free Crafting of Scoring Functions using Constraints Based on Domain Expertise

Ragja Palakkadavath, Sarath Sivaprasad, Shirish Karande, Niranjan PedanekararXiv 2022 · arXiv

TL;DR

Crafting quantitative scoring functions (like university rankings or ad performance scores) usually requires tedious trial-and-error or hard-to-get labeled data. This paper introduces a Label-Free Weakly Supervised (LFWS) approach that trains a neural network using only qualitative domain constraints (e.g., "higher click rate should increase the score") converted into differentiable loss functions. The result is an explainable, data-driven scoring system built entirely without explicit labels.

What Problem Does This Paper Address?

In many industries, decision-makers need a single number to summarize complex performance metrics—think employee ratings, movie popularity, or ad campaign effectiveness. However, creating these scoring functions is notoriously difficult. Labeled data is rarely available, human annotation introduces personal biases, and asking experts to assign exact mathematical weights (e.g., "give impressions exactly 20% importance") results in arbitrary, ad-hoc guesses. Consequently, experts spend immense time in a frustrating loop of trial-and-error to get intuitive score distributions.

The authors address this gap by creating a framework that allows domain experts to inject their qualitative insights directly into a machine learning model, bypassing the need for manual data labeling or precise weight guessing.

Methods

The authors propose the Label-Free Weakly Supervised (LFWS) framework. Instead of learning from ground-truth labels, a 3-layer dense neural network learns directly from domain-specified mathematical properties encoded as loss functions:

  • Monotonicity Constraint: Enforces that specific features only affect the score in one direction (positively or negatively). This is achieved by forcing network weights to be positive via log-domain learning.
  • Boundedness & Mode Constraints: Uses a modified hinge loss to keep scores within a desired range $[a, b]$ and a mode loss to center scores around a specific value.
  • Sensitivity Constraint: A novel loss function based on partial derivatives ($\frac{\partial f(x)}{\partial x_i}$) that prioritizes the relative impact of important features over others without requiring explicit numerical weights.
  • Output Constraint: Minimizes the KL divergence between the generated score distribution and a target distribution (e.g., a Gaussian bell curve or an exponential distribution).

Datasets

The framework was validated across five datasets:

  • Synthetic Dataset: 10,000 data points with 4 Gaussian features and a known mathematical ground truth to benchmark baseline performance.
  • IMDB Dataset: ~6,000 movies scored on features like votes, budget, and gross income.
  • CWUR Dataset: Center for World University Rankings data, used to compare LFWS against supervised models using actual ranking scores.
  • Journal Rank Dataset: Impact factor ratings for the top 1,000 journals in 2019.
  • Ad Campaign Dataset: A proprietary dataset containing 450 LinkedIn lead-generation ads, simulating a real-world deployment scenario without ground-truth labels.

Key Insights and Ideas

  • Monotonicity is the Bedrock: Without the monotonicity constraint, the unsupervised network completely fails to learn meaningful scoring functions, even when all other distribution and boundary losses are active. It acts as a powerful regularizer in the absence of labels.
  • Relative Sensitivity Trumps Exact Weights: Experts don't need to guess exact percentages for feature importance. Simply stating that "Feature A should matter more than Feature B" provides enough gradient guidance for the network to align its internal weights effectively.
  • Distribution Shaping Works: By incorporating KL divergence into the loss function, the model can naturally force scores into realistic distributions (like a standard performance bell curve) without sacrificing feature correlations.

Key Results

  • When compared against a fully supervised XGBoost model (which had access to ground-truth labels), the label-free LFWS model performed remarkably well. On the CWUR dataset, LFWS achieved a Spearman rank correlation of 0.90 compared to XGBoost's 0.98.
  • On the Journal Rank dataset, LFWS reached a rank correlation of 0.76 (XGBoost scored 0.81).
  • Ablation studies demonstrated that adding the sensitivity loss significantly boosted the Spearman correlation ($\rho$) for designated key features (e.g., Form Fill Rate in the Ad dataset jumped from 0.480 to 0.887).
  • As shown in the kernel density estimate plots, the LFWS model successfully matched the target modes and distributions specified by the domain constraints.
Figure 2: Kernel density estimates for generated scores

Limitations and Future Work

  • Hyperparameter Sensitivity: Balancing the weights ($\alpha, \beta, \gamma$) between the different constraint losses currently requires manual tuning, which reintroduces a bit of the trial-and-error the paper seeks to avoid.
  • Future Automation: The authors plan to develop methods to automatically tweak these loss weights based on high-level, simple feedback from domain experts (e.g., "the model hasn't learned enough sensitivity for feature X").
  • Validation Tools: They also aim to build low-effort validation interfaces, such as interactive visualizations and pairwise comparison tools, to help experts easily verify and refine the generated scoring functions in real-world deployments.

Summary updated

Abstract

Several real-life applications require crafting concise, quantitative scoring functions (also called rating systems) from measured observations. For example, an effectiveness score needs to be created for advertising campaigns using a number of engagement metrics. Experts often need to create such scoring functions in the absence of labelled data, where the scores need to reflect business insights and rules as understood by the domain experts. Without a way to capture these inputs systematically, this becomes a time-consuming process involving trial and error. In this paper, we introduce a label-free practical approach to learn a scoring function from multi-dimensional numerical data. The approach incorporates insights and business rules from domain experts in the form of easily observable and specifiable constraints, which are used as weak supervision by a machine learning model. We convert such constraints into loss functions that are optimized simultaneously while learning the scoring function. We examine the efficacy of the approach using a synthetic dataset as well as four real-life datasets, and also compare how it performs vis-a-vis supervised learning models.
BibTeX
@misc{palakkadavath2022i,
  author = {Ragja Palakkadavath and Sarath Sivaprasad and Shirish Karande and Niranjan Pedanekar},
  title = {{I Know Therefore I Score: Label-Free Crafting of Scoring Functions using Constraints Based on Domain Expertise}},
  year = {2022},
  eprint = {2203.10085},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.10085},
}
Figure