Bias Impact Analysis of AI in Consumer Mobile Health Technologies: Legal, Technical, and Policy

Kristine Joan Kristine Gloria, Nidhi Rastogi, Stevie DeGroffarXiv 2022 · DOI · OpenReview

Introduction

The rapid proliferation of consumer mobile health (mHealth) technologies—ranging from fitness wearables to mental health diagnostic apps—has been accelerated by the COVID-19 pandemic and the ubiquity of mobile sensors. By 2025, the mHealth market is projected to reach $600 billion. However, this technological surge has outpaced existing ethical and legal guardrails, leading to a critical regulatory "HIPAA Gap." In the United States, the Health Insurance Portability and Accountability Act (HIPAA) primarily protects data handled by "covered entities" like hospitals and clinics. Most consumer mHealth apps fall outside this jurisdiction, operating in a self-regulated environment where algorithmic bias can manifest in diagnostics or behavioral interventions without clinical scrutiny.

Mapping the Lifecycle of Bias

Algorithmic bias in mHealth is not a singular technical error but a multifaceted issue that permeates the entire development lifecycle. To address this, we must identify specific entry points where unwanted bias is introduced across data collection, system design, and real-world deployment.

Entry points for unwanted bias across the data, design, and deployment phases of mHealth AI.

As illustrated above, bias begins at the Data stage through inaccurate user inputs or sampling bias that lacks representative data. In the Design phase, non-inclusive model building and non-active feedback loops can codify existing power dynamics. Finally, during Deployment, inequitable distribution of devices and hazardous repurposing of expert systems can generate non-representative feedback, reinforcing systemic health disparities.

The Invisibility of Exclusion: A COVID-19 Case Study

The dangers of selection bias are most evident in public health responses that rely on digital proxies. During the COVID-19 pandemic, many tracking and modeling efforts utilized GPS and Call Detail Records (CDRs) to monitor mobility and proximity. However, this approach creates a "funnel of exclusion."

The funnel of exclusion: how reliance on smartphone data misrepresents the broader population.

When public health models rely exclusively on smartphone data, they inadvertently ignore populations with low smartphone penetration, such as the elderly, those in extreme poverty, or individuals without stable housing. This selection bias makes marginalized communities invisible to the public health response, intensifying existing inequities. Data, in this context, becomes a codification of power, deciding whose health needs are prioritized based on their digital footprint.

Frameworks for Mitigation and Regulatory Gaps

Mitigating these harms requires more than simple technical fixes like dataset re-weighting. We advocate for the SMACTR framework (Scoping, Mapping, Artifact Collection, Testing, and Reflection), which treats AI auditing as a continuous process similar to safety audits in the aerospace industry. This end-to-end auditing ensures that bias is addressed from initial scoping to post-deployment reflection.

From a regulatory perspective, the landscape is shifting. The European Union’s Artificial Intelligence Act (AIA) represents a first-mover advantage in regulating high-risk AI. In contrast, the U.S. approach remains fragmented, with the FDA only recently pivoting toward "Pre-Cert" programs for software-based medical devices. Without comprehensive oversight, mHealth harms can be categorized into three primary buckets:

  1. Data Privacy: Breaches of personally identifiable information and misuse of sensitive health data.
  2. Inaccuracy: Inaccurate diagnostics or behavioral interventions that lead to a loss of patient agency.
  3. Inequity and Access: The propagation of biased sampling that limits health and digital literacy for underserved groups.

Conclusion and Future Outlook

While technical toolkits like IBM’s AI Fairness 360 provide a starting point, the path toward equitable mHealth AI faces significant economic and structural hurdles. Comprehensive third-party auditing is expensive, and it remains unclear who will bear these costs in a competitive consumer market. Furthermore, the U.S. multi-payer healthcare system lacks financial incentives for tech creators and health professionals to collaborate on bias reduction. Moving forward, we must move beyond the "single bullet" fallacy and adopt iterative, multi-stakeholder assessments to ensure that the AI in our pockets empowers all patients equitably.

Summary updated

Abstract

Today's large-scale algorithmic and automated deployment of decision-making systems threatens to exclude marginalized communities. Thus, the emergent danger comes from the effectiveness and the propensity of such systems to replicate, reinforce, or amplify harmful existing discriminatory acts. Algorithmic bias exposes a deeply entrenched encoding of a range of unwanted biases that can have profound real-world effects that manifest in domains from employment, to housing, to healthcare. The last decade of research and examples on these effects further underscores the need to examine any claim of a value-neutral technology. This work examines the intersection of algorithmic bias in consumer mobile health technologies (mHealth). We include mHealth, a term used to describe mobile technology and associated sensors to provide healthcare solutions through patient journeys. We also include mental and behavioral health (mental and physiological) as part of our study. Furthermore, we explore to what extent current mechanisms - legal, technical, and or normative - help mitigate potential risks associated with unwanted bias in intelligent systems that make up the mHealth domain. We provide additional guidance on the role and responsibilities technologists and policymakers have to ensure that such systems empower patients equitably.
BibTeX
@misc{gloria2022bias,
  author = {Kristine Joan Kristine Gloria and Nidhi Rastogi and Stevie DeGroff},
  title = {{Bias Impact Analysis of AI in Consumer Mobile Health Technologies: Legal, Technical, and Policy}},
  year = {2022},
  eprint = {2209.05440},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.05440},
}
Figure