Impact of Predicting Health Care Utilization Via Web Search Behavior: A Data-Driven Analysis

Vibhu Agarwal, Liangliang Zhang, Josh Zhu, Shiyuan Fang, T. S. Cheng, Chloe Hong, Nigam H. ShahJournal of Medical Internet Research 2016 · DOI

TL;DR

This paper explores the use of geotagged mobile search logs to predict future health care utilization, demonstrating that such predictions can indeed be accurate and valuable for healthcare providers. By analyzing search behavior, the study shows a significant increase in advertisement conversion rates when ads are targeted to individuals predicted to need healthcare services, highlighting the practical utility of this approach.

What Problem Does This Paper Address?

The rising cost of healthcare in the US has left millions without access to necessary services. To improve efficiency and accessibility, healthcare providers need better ways to understand and predict health care utilization at a population level. While previous studies have suggested that mining internet search logs can reveal patterns of health care resource use, the practical utility of these predictions for various stakeholders in the healthcare ecosystem remained unclear. This paper aims to bridge that gap by evaluating the utility of health care use predictions, specifically from the perspective of healthcare providers, using a novel closed-loop evaluation method.

Methods

The researchers developed a system to predict future health care utilization by analyzing deidentified mobile internet search logs. Key methodological components include:

  • Geofencing and Labeling: Users' search logs were analyzed to infer their presence within the geofence of a medical facility. Search logs were then assigned "medical facility visit" labels based on location and duration information.
  • Feature Engineering: A comprehensive set of features was engineered, categorized into:
  • General Features: Basic attributes like number of searches, session duration, and search text length.
  • Semantic Features: Analyzing the content of search queries, including searches for diseases, drugs, and medical procedures, using a large biomedical terminology.
  • Location Features: Incorporating geographical information from searches, including proximity to landmarks and specific location categories.
  • Daywise and Aggregate Features: Features were extracted for each individual search day within an analysis window and also aggregated over the entire window.
  • Predictive Modeling: Various machine learning models were trained, including Lasso, Ridge, Elastic Net, Support Vector Machines (SVM), and Random Forests. The Random Forests model with daywise features showed the best performance.
  • Utility Evaluation: The utility of the predictions was evaluated using a two-pronged approach:
  • Offline Evaluation: Analyzing historical data to assess the correlation between predicted utilization scores and actual conversion rates.
  • Online Evaluation: Conducting an A/B test within Baidu's mobile search ads system, where advertisements for medical facilities were targeted to users with high predicted utilization scores. The impact on ad show conversion rates and cost per action (CPA) was measured.

Datasets

The study utilized a large dataset of deidentified mobile internet search logs from Baidu, comprising over 1 billion searches from 9.5 million users between March and May 2015. This dataset was collected from users in China. The researchers also used a collaborative research program to access this data.

Key Insights and Ideas

  • Search Behavior as a Predictor: The study strongly suggests that patterns in web search behavior, especially when combined with location data, can accurately predict future health care utilization.
  • The Power of Location: Omitting location features significantly degraded the model's predictive performance, highlighting the crucial role of geographical context in understanding health-seeking behavior.
  • Daywise Features Matter: Analyzing search behavior on a day-by-day basis (daywise features) proved more effective than using aggregated features over longer periods, indicating that the temporal progression of search activities is key.
  • Practical Utility for Advertisers: The online experiment demonstrated that targeting health care advertisements to individuals predicted to need services leads to a tangible increase in ad conversion rates, proving the real-world utility of these predictions for healthcare providers and advertisers.
  • Anxiety as a Signal: Features like the "interval reduction score" (measuring the shortening/lengthening of intervals between health-related searches) and "mean information content score" (indicating search specificity) were identified as important predictors, potentially reflecting user anxiety.

Key Results

  • Model Performance: The Random Forests model using daywise features achieved the highest Area Under the Curve (AUC) of 0.796 for predicting medical visits.
  • Feature Importance: The omission of location features caused the largest drop in AUC, followed by semantic and general features, underscoring their importance.
  • Online Utility: The online evaluation showed a 3.96% increase in the show conversion rate for users targeted with health care advertisements based on predicted utilization scores, compared to a control group. This indicates a more efficient use of advertising budgets.
  • Cost Reduction: The model with location-based features also demonstrated a lower Cost Per Action (CPA) in the online experiment.

Limitations and Future Work

  • Data Sparsity and Noise: Geotagged search data is inherently sparse and prone to noise, including false positives and negatives due to labeling challenges. The researchers acknowledge that it's not possible to completely eliminate these issues.
  • Labeling Accuracy: Assigning medical facility visit labels solely based on geofences is imperfect. While efforts were made to filter "weak" labels, some inaccuracies may persist.
  • Dynamic Calibration: The paper suggests that further investigation into dynamic calibration of decision rules for mapping prediction scores to advertisement serving is needed to optimize utility.
  • Generalizability: While the study used a large dataset, it was specific to Baidu search logs from China. Future work could explore generalizability across different search engines and cultural contexts.
  • Causality: The paper acknowledges that establishing direct causality between ad display and hospitalization is complex and beyond the scope of the current study.

The authors suggest that future work could explore methods that learn from only positive labeled data to control for false negatives and investigate methods that attempt to learn from larger observation windows to create clearer feature sets.

Summary updated

Abstract

Results from our experiments done in a research setting suggest that it is possible to accurately predict future patient visits from geotagged mobile search logs. Results from the offline and online experiments on the utility of health utilization predictions suggest that such prediction can have utility for health care providers.
BibTeX
@article{agarwal2016impact,
  author = {Vibhu Agarwal and Liangliang Zhang and Josh Zhu and Shiyuan Fang and T. S. Cheng and Chloe Hong and Nigam H. Shah},
  title = {{Impact of Predicting Health Care Utilization Via Web Search Behavior: A Data-Driven Analysis}},
  journal = {Journal of Medical Internet Research},
  year = {2016},
  doi = {10.2196/jmir.6240},
}

Loading related content...

Figure