Using Health Concept Surveying to Elicit Usable Evidence: Case Studies of a Novel Evaluation Methodology (Preprint)

Alex Mariakakis, Ravi Karkar, Shwetak Patel, Julie A. Kientz, James Fogarty, Sean A. Munson2021 · DOI

TL;DR

This paper introduces "health concept surveying," a novel method for evaluating early-stage health technology designs. By using scenario-based surveys and analyzing responses with path analysis, it helps researchers understand how design choices impact adoption and acceptability, generating usable evidence without needing a working prototype.

What Problem Does This Paper Address?

Developing effective health intervention technologies (HITs) is challenging. While rapid prototyping is common for early feedback, it often doesn't yield "usable evidence" – insights into the causal mechanisms behind user adoption and acceptance. This means developers might get feedback on what users like, but not why or how specific design decisions truly influence behavior. Deploying HITs too early can also pose risks to users. The authors aim to bridge this gap by providing a method that generates deeper, causal understanding of HIT design's impact on user behavior, even before a functional prototype exists.

Methods

The core contribution is the health concept surveying method. It's a variation of vignette testing where participants respond to scenarios describing a health issue and a conceptual HIT. The survey instrument is structured in four stages:

  1. Preintervention: Participants read a scenario and answer questions based on the Health Belief Model (HBM) to establish baseline perceptions (e.g., perceived seriousness, susceptibility, benefits, barriers).
  2. Intervention: Participants are presented with a description of the HIT, potentially with manipulated design features (e.g., an incentive, visual explanations).
  3. Postintervention: Participants re-evaluate their HBM constructs and indicate their intention to use the HIT (TechnologyInterest) and their willingness to take action (ActionChange).
  4. End of Survey: Participants provide demographic and other "modifying variables" that can influence their responses.

Responses are analyzed using path analysis, a statistical technique that models causal relationships between variables. This allows the researchers to untangle how specific HIT design elements and user characteristics influence adoption and acceptability.

Datasets

The paper does not introduce new datasets for training models. Instead, it presents three case studies using data collected from participants recruited via platforms like Facebook, Reddit, and university mailing lists. The participants were screened for age and smartphone ownership.

  • Case Study 1: N=54, focused on a skin cancer screening app with an incentive.
  • Case Study 2: N=54, focused on a skin cancer screening app with visual explanations of results.
  • Case Study 3: N=263, explored trade-offs between sensitivity and specificity across three different medical scenarios (pink eye, skin cancer, halitosis).

Key Insights and Ideas

  • Beyond Preferences: Health concept surveying moves beyond simply asking users what they prefer in a design. It aims to uncover the causal pathways through which design choices influence behavior, offering deeper insights than traditional usability testing.
  • Early-Stage Value: The method is particularly valuable for early-stage design because it doesn't require a fully functional prototype. This allows designers to test concepts and gather evidence before investing significant development resources.
  • HBM as a Framework: Leveraging the Health Belief Model (HBM) provides a structured way to measure proximal cognitive factors that mediate the relationship between HIT design and user outcomes. This allows for a more systematic analysis of "why" users might adopt or reject a technology.
  • Actionable Evidence: The method is designed to produce "usable evidence" that can directly inform design decisions and contribute to broader theories of HIT adoption and acceptability.

Key Results

  • Case Study 1 (Incentive): A wait time incentive for dermatologists showed a strong positive effect on participants' willingness to take action after a positive skin cancer test result, primarily by reducing perceived barriers. However, many participants indicated they would act even without the incentive, suggesting other HBM factors are also crucial.
  • Case Study 2 (Visuals): Visual explanations of algorithmic decisions increased trust in negative test results but did not significantly impact participants' decision-making. The perceived seriousness of skin cancer was a stronger driver for action (or inaction) than the visual explanations.
  • Case Study 3 (Accuracy): Participants prioritized overall accuracy (sensitivity and specificity) in health screening apps, especially for common and stigmatizing conditions. They were willing to use apps regardless of their perception of the medical condition's threat, as long as the app was accurate. The importance of sensitivity vs. specificity varied by condition type.

Limitations and Future Work

  • Intention-Behavior Gap: The authors acknowledge that stated intentions in hypothetical scenarios may not always translate to real-world behavior. Future work could explore ways to increase the realism of scenarios or measure the degree to which respondents resonate with them.
  • Scenario Realism: Ensuring scenarios are plausible and realistic is crucial. The authors used a pilot study for validation and suggest future researchers do the same, especially when investigating high-level questions.
  • Focus on Short-Term Actions: The authors suggest future studies might benefit from focusing on short-term actions rather than long-term goals, as intention is a weaker predictor for the latter.
  • Other Design Factors: The paper explored incentives, result presentation, and accuracy. Future work could investigate other design decisions like pricing, endorsements, and the impact of different intervention strategies.
  • Complementary Methods: Health concept surveying is seen as complementary to qualitative methods like focus groups, offering scalability and systematic analysis of variables. The authors encourage exploring how psychological frameworks can be integrated into other evaluation techniques.

Summary updated

Abstract

<sec> <title>BACKGROUND</title> Developers, designers, and researchers use rapid prototyping methods to project the adoption and acceptability of their health intervention technology (HIT) before the technology becomes mature enough to be deployed. Although these methods are useful for gathering feedback that advances the development of HITs, they rarely provide usable evidence that can contribute to our broader understanding of HITs. </sec> <sec> <title>OBJECTIVE</title> In this research, we aim to develop and demonstrate a variation of vignette testing that supports developers and designers in evaluating early-stage HIT designs while generating usable evidence for the broader research community. </sec> <sec> <title>METHODS</title> We proposed a method called &lt;i&gt;health concept surveying&lt;/i&gt; for untangling the causal relationships that people develop around conceptual HITs. In health concept surveying, investigators gather reactions to design concepts through a scenario-based survey instrument. As the investigator manipulates characteristics related to their HIT, the survey instrument also measures proximal cognitive factors according to a health behavior change model to project how HIT design decisions may affect the adoption and acceptability of an HIT. Responses to the survey instrument were analyzed using path analysis to untangle the causal effects of these factors on the outcome variables. </sec> <sec> <title>RESULTS</title> We demonstrated health concept surveying in 3 case studies of sensor-based health-screening apps. Our first study (N=54) showed that a wait time incentive could influence more people to go see a dermatologist after a positive test for skin cancer. Our second study (N=54), evaluating a similar application design, showed that although visual explanations of algorithmic decisions could increase participant trust in negative test results, the trust would not have been enough to affect people’s decision-making. Our third study (N=263) showed that people might prioritize test specificity or sensitivity depending on the nature of the medical condition. </sec> <sec> <title>CONCLUSIONS</title> Beyond the findings from our 3 case studies, our research uses the framing of the Health Belief Model to elicit and understand the intrinsic and extrinsic factors that may affect the adoption and acceptability of an HIT without having to build a working prototype. We have made our survey instrument publicly available so that others can leverage it for their own investigations. </sec>
BibTeX
@misc{mariakakis2021using,
  author = {Alex Mariakakis and Ravi Karkar and Shwetak Patel and Julie A. Kientz and James Fogarty and Sean A. Munson},
  title = {{Using Health Concept Surveying to Elicit Usable Evidence: Case Studies of a Novel Evaluation Methodology (Preprint)}},
  year = {2021},
  doi = {10.2196/preprints.30474},
}
Figure