A Mathematical Programming-Based Approach to Determining Objective Functions from Qualitative and Subjective Comparisons

Takayuki YoshizumiAAAI 2015 · DOI

Introduction

In optimization and simulation, defining an accurate objective function—a mathematical formula that ranks the quality of solutions—is a critical yet challenging task. Typically, these functions are constructed as weighted sums of various metrics. While domain experts excel at making qualitative comparisons (e.g., "Solution A is significantly better than B"), they often struggle to provide consistent absolute numerical scores. Furthermore, different experts possess unique internal "rulers," where one's "far better" might be another's "better."

Traditional machine learning (ML) approaches, such as learning-to-rank or Support Vector Machines (SVMs), often fail to capture these nuanced degrees of difference and individual subjectivity. This paper introduces a mathematical programming approach that treats subjective evaluations as simultaneous linear inequalities with learnable boundaries, effectively translating qualitative human intuition into precise mathematical models.

Methodology

The proposed framework shifts away from standard ML loss-minimization toward Mathematical Programming, specifically utilizing Linear Programming (LP) and Mixed Integer Programming (MIP). The core innovation rests on two pillars:

  1. Ordinal Differences: Rather than binary comparisons ($A > B$), the model captures the magnitude of preference. If an expert designates one pair as "far better" and another as merely "better," the model enforces a constraint ensuring the mathematical gap in the first pair exceeds that of the second.
  2. Subjective Boundaries: The model introduces decision variables to represent the specific thresholds for each expert. This allows the system to independently learn how different individuals define categories like "similar," "better," or "far better."
Boundary estimation for three experts showing the model's ability to learn individual subjective thresholds.

By formulating the problem as an LP, the system finds weights that minimize total error across all comparisons. The extension to MIP allows for automatic selection of optimal basis functions (e.g., linear vs. non-linear terms) and feature selection through regularization.

Experimental Setup

The approach was evaluated using a synthetic dataset to maintain controlled conditions:

  • Experts: Three simulated experts with distinct internal boundaries.
  • Metrics: Ten objective terms with predefined "correct" weights of 0.1 each.
  • Data: 500 pairs of solutions generated via a normal distribution, including random noise to simulate human inconsistency.
  • Baselines: The method was compared against a baseline that ignores degrees of difference (Baseline 1) and an $\epsilon$-insensitive SVM-like baseline that assumes uniform boundaries across experts (Baseline 2).

Results and Discussion

The proposed method demonstrated superior performance in recovering the true weights of the objective function. With 500 ratings, the method achieved a relative error of 1.76%, significantly lower than the 5.20% error observed in Baseline 2.

Relative error comparison between the proposed method and two baselines as a function of data volume.

Key findings include:

  • Subjectivity as a Signal: By treating individual subjectivity as a learnable parameter rather than noise, the model extracts more information from the same dataset.
  • Data Efficiency: While the model requires more data initially to calibrate the extra boundary variables, its accuracy improves more rapidly than simpler ranking algorithms as the data volume increases.
  • Information Extraction: The use of an ordinal scale for differences provides a higher resolution of the underlying weight distribution compared to standard ranking methods.

Conclusion

This research highlights the power of mathematical programming in handling subjective, qualitative data. By explicitly modeling expert boundaries, the system achieves high precision in weight estimation. Future work will focus on automating the discovery of basis functions and validating the model with real-world human experts in domains such as industrial optimization and game balancing.

Summary updated

BibTeX
@inproceedings{yoshizumi2015mathematical,
  author = {Takayuki Yoshizumi},
  title = {{A Mathematical Programming-Based Approach to Determining Objective Functions from Qualitative and Subjective Comparisons}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i1.9567},
}
Figure