TL;DR
This paper introduces "health concept surveying," a novel method for evaluating early-stage health technology designs. By using scenario-based surveys and analyzing responses with path analysis, it helps researchers understand how design choices impact adoption and acceptability, generating usable evidence without needing a working prototype.
What Problem Does This Paper Address?
Developing effective health intervention technologies (HITs) is challenging. While rapid prototyping is common for early feedback, it often doesn't yield "usable evidence" – insights into the causal mechanisms behind user adoption and acceptance. This means developers might get feedback on what users like, but not why or how specific design decisions truly influence behavior. Deploying HITs too early can also pose risks to users. The authors aim to bridge this gap by providing a method that generates deeper, causal understanding of HIT design's impact on user behavior, even before a functional prototype exists.
Methods
The core contribution is the health concept surveying method. It's a variation of vignette testing where participants respond to scenarios describing a health issue and a conceptual HIT. The survey instrument is structured in four stages:
- Preintervention: Participants read a scenario and answer questions based on the Health Belief Model (HBM) to establish baseline perceptions (e.g., perceived seriousness, susceptibility, benefits, barriers).
- Intervention: Participants are presented with a description of the HIT, potentially with manipulated design features (e.g., an incentive, visual explanations).
- Postintervention: Participants re-evaluate their HBM constructs and indicate their intention to use the HIT (TechnologyInterest) and their willingness to take action (ActionChange).
- End of Survey: Participants provide demographic and other "modifying variables" that can influence their responses.
Responses are analyzed using path analysis, a statistical technique that models causal relationships between variables. This allows the researchers to untangle how specific HIT design elements and user characteristics influence adoption and acceptability.
Datasets
The paper does not introduce new datasets for training models. Instead, it presents three case studies using data collected from participants recruited via platforms like Facebook, Reddit, and university mailing lists. The participants were screened for age and smartphone ownership.
- Case Study 1: N=54, focused on a skin cancer screening app with an incentive.
- Case Study 2: N=54, focused on a skin cancer screening app with visual explanations of results.
- Case Study 3: N=263, explored trade-offs between sensitivity and specificity across three different medical scenarios (pink eye, skin cancer, halitosis).
Key Insights and Ideas
- Beyond Preferences: Health concept surveying moves beyond simply asking users what they prefer in a design. It aims to uncover the causal pathways through which design choices influence behavior, offering deeper insights than traditional usability testing.
- Early-Stage Value: The method is particularly valuable for early-stage design because it doesn't require a fully functional prototype. This allows designers to test concepts and gather evidence before investing significant development resources.
- HBM as a Framework: Leveraging the Health Belief Model (HBM) provides a structured way to measure proximal cognitive factors that mediate the relationship between HIT design and user outcomes. This allows for a more systematic analysis of "why" users might adopt or reject a technology.
- Actionable Evidence: The method is designed to produce "usable evidence" that can directly inform design decisions and contribute to broader theories of HIT adoption and acceptability.
Key Results
- Case Study 1 (Incentive): A wait time incentive for dermatologists showed a strong positive effect on participants' willingness to take action after a positive skin cancer test result, primarily by reducing perceived barriers. However, many participants indicated they would act even without the incentive, suggesting other HBM factors are also crucial.
- Case Study 2 (Visuals): Visual explanations of algorithmic decisions increased trust in negative test results but did not significantly impact participants' decision-making. The perceived seriousness of skin cancer was a stronger driver for action (or inaction) than the visual explanations.
- Case Study 3 (Accuracy): Participants prioritized overall accuracy (sensitivity and specificity) in health screening apps, especially for common and stigmatizing conditions. They were willing to use apps regardless of their perception of the medical condition's threat, as long as the app was accurate. The importance of sensitivity vs. specificity varied by condition type.
Limitations and Future Work
- Intention-Behavior Gap: The authors acknowledge that stated intentions in hypothetical scenarios may not always translate to real-world behavior. Future work could explore ways to increase the realism of scenarios or measure the degree to which respondents resonate with them.
- Scenario Realism: Ensuring scenarios are plausible and realistic is crucial. The authors used a pilot study for validation and suggest future researchers do the same, especially when investigating high-level questions.
- Focus on Short-Term Actions: The authors suggest future studies might benefit from focusing on short-term actions rather than long-term goals, as intention is a weaker predictor for the latter.
- Other Design Factors: The paper explored incentives, result presentation, and accuracy. Future work could investigate other design decisions like pricing, endorsements, and the impact of different intervention strategies.
- Complementary Methods: Health concept surveying is seen as complementary to qualitative methods like focus groups, offering scalability and systematic analysis of variables. The authors encourage exploring how psychological frameworks can be integrated into other evaluation techniques.
Summary updated