Design of reliable technology valuation model with calibrated machine learning of patent indicators

Seunghyun Lee, Janghyeok Yoon, Jaewoong ChoiarXiv 2024 · DOI · arXiv

Abstract

In the domain of intellectual property management, determining the economic value of patents is a critical yet complex task. While machine learning (ML) models have demonstrated high predictive accuracy, they often suffer from overconfidence, creating a "trust gap" for decision-makers. This paper proposes a framework for patent valuation that prioritizes model reliability through calibration and interpretability. Using a dataset of 46,973 semiconductor patents, we evaluate four ML architectures—Logistic Regression, Random Forest, Neural Networks, and XGBoost—using Expected Calibration Error (ECE) alongside standard performance metrics. Our findings indicate that an optimized Random Forest model provides the best balance between predictive power and calibrated reliability. Furthermore, SHAP analysis reveals that the drivers of patent value shift significantly depending on the model's confidence level, highlighting the importance of context in automated valuation.

Introduction

The valuation of technological assets is a cornerstone of strategic business decisions, mergers and acquisitions, and litigation. Traditional valuation methods often rely on expert qualitative assessment, which is time-consuming and difficult to scale. While machine learning has emerged as a solution to automate this process, most existing models function as "black boxes." These models may achieve high accuracy but often lack reliability; specifically, their predicted probabilities are frequently uncalibrated, leading to overconfident predictions that do not reflect true success rates.

To address this, we present a framework that shifts the focus from "maximum accuracy" to "calibrated reliability." By ensuring that a model’s predicted probability (e.g., a 90% chance of value) aligns with the actual observed frequency of valuable patents, we can bridge the trust gap for industry practitioners.

Methodology

Our proposed framework consists of four integrated stages designed to ensure both performance and transparency.

1. Indicator Extraction and Proxy Selection

We identified 50 quantitative patent indicators categorized into six dimensions: scope and coverage, priority, completeness, development effort, technology environment, and prior knowledge. To circumvent the volatility of citation-based metrics across different technological fields, we utilized patent maintenance—the duration for which an owner pays fees to keep a patent active—as a domain-agnostic proxy for economic value.

2. Data Preprocessing

The dataset comprised 46,973 semiconductor patents (IPC H01L) from the USPTO (2000–2019). Given that the majority of patents are not maintained for their full legal term, the dataset was inherently imbalanced. We applied Tomek Links, an under-sampling technique that removes ambiguous data points near the decision boundary, thereby enhancing the model's ability to distinguish between high-value and low-value patents.

3. Model Calibration and Evaluation

We trained and compared four models: Logistic Regression (LR), Random Forest (RF), Neural Networks (NN), and XGBoost (XGB). Beyond standard metrics like the F1-score and Matthews Correlation Coefficient (MCC), we utilized Expected Calibration Error (ECE). ECE measures the discrepancy between predicted probabilities and actual outcomes, serving as a primary metric for model reliability.

4. Interpretability via SHAP

To understand the underlying logic of the predictions, we employed SHAP (SHapley Additive exPlanations). Uniquely, we analyzed feature importance across different "confidence bins" (e.g., 0–20% vs. 80–100% probability), allowing us to see how the drivers of value change as the model becomes more certain.

Results

All evaluated models demonstrated strong baseline performance, with accuracies ranging between 90% and 91%. However, their reliability profiles varied significantly:

  • XGBoost and Random Forest were the top performers in terms of predictive power. XGBoost showed excellent balance between sensitivity and specificity.
  • Calibration Performance: Logistic Regression exhibited notable overconfidence, particularly in the lower probability ranges. In contrast, XGBoost and Random Forest remained closely aligned with the "perfect calibration" diagonal.
  • Model Selection: By mapping the models on a Pareto-front (ECE vs. MCC), the optimized Random Forest was identified as the superior choice for practical application, offering the most robust trade-off between being accurate and being "honest" about its uncertainty.

Discussion: The Dynamics of Patent Value

The SHAP analysis provided critical insights into the "why" behind patent value. We observed that the importance of specific indicators is context-dependent:

  • High-Confidence Drivers: In scenarios where the model was highly certain of a patent's value, the scale and activity level of the technological field (indicators TE_1 and TE_2) were the strongest positive predictors.
  • The Red Queen Effect: In low-confidence regions, high competitiveness within a field (TE_3) often had a negative impact. This suggests that in hyper-competitive or saturated markets, even technically sound patents may face diminished economic returns due to the density of competing intellectual property—a phenomenon akin to the "Red Queen" hypothesis where one must run just to stay in place.

Conclusion

This research demonstrates that high predictive accuracy is insufficient for industrial technology valuation; reliability and interpretability are equally paramount. By integrating calibration techniques and SHAP-based confidence binning, we provide a framework that not only identifies valuable patents but also quantifies the trustworthiness of those identifications. Future work will focus on expanding this framework to other technological domains, such as biotechnology, and incorporating additional reliability metrics like the Brier score to further refine the model's "honesty."

Summary updated

BibTeX
@misc{lee2024design,
  author = {Seunghyun Lee and Janghyeok Yoon and Jaewoong Choi},
  title = {{Design of reliable technology valuation model with calibrated machine learning of patent indicators}},
  year = {2024},
  eprint = {2406.05446},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.05446},
}
Figure