Food for thought: Ethical considerations of user trust in computer vision

Kaylen J. Pfisterer, Jennifer Boger, Alexander WongarXiv 2019 · arXiv

Introduction

Malnutrition represents a significant yet often silent crisis in long-term care (LTC) facilities, affecting approximately 44% of older adult residents. While monitoring food and fluid intake is a clinical mandate, current practices rely heavily on manual observations recorded by caregivers. This "gold standard" is frequently criticized for being subjective, retrospective, and prone to inaccuracies. While computer vision (CV) offers a promising solution for objective, automated tracking, its deployment in sensitive environments necessitates a deep dive into the ethical implications of user trust. This paper explores the development of the AFINI-T system and the critical need to manage "over-trust" and automation bias among clinical staff.

The AFINI-T System

The researchers developed AFINI-T (Automated Food Imaging and Nutrient Intake Tracking), an end-to-end system designed to automate the tracking process through four cascading CV subsystems:

  1. Optical Acquisition: Utilizing an integrated RGB-D (depth) camera to capture high-fidelity images.
  2. Food Segmentation: Separating food items from the plate and background environment.
  3. Food Volume Estimation: Leveraging depth data to calculate the precise volume of identified segments.
  4. Nutrient Estimation: Inferring nutritional values based on volume and food type to provide a comprehensive intake summary.

Unlike general-purpose apps, AFINI-T benefits from the constrained environment of LTC facilities, which feature regulated portion sizes and set menus, making the CV task more tractable.

Methodology

The study employed a 6-stage iterative participatory design sprint, engaging 38 stakeholders, including personal support workers, nurses, and dietitians. This collaborative approach aimed to move from initial ideation to a "Goldilocks" quality horizontal prototype. To evaluate the human element, trust was quantified using a modified version of the scale developed by Jian et al. The AFINI-T prototype was compared against the existing paper-based charting system across several dimensions, including deceptiveness, wariness, and confidence.

Results

The quantitative evaluation revealed a striking preference for the automated prototype over the established manual system. The AFINI-T prototype achieved a high System Usability Scale (SUS) score of 89.2. Statistical analysis of trust metrics showed significant improvements in user perception across several key indicators.

Table 1: Comparison of trust metrics between the existing manual system and the AFINI-T prototype.

As shown in Table 1, only 17% of participants perceived AFINI-T as "deceptive," compared to 45% for the existing system ($p = .005$). Notably, 0% of users reported feeling "wary" of the AFINI-T prototype, whereas 50% expressed wariness toward the manual system ($p = .006$). Furthermore, 83% of participants felt confident in the AFINI-T system, a stark contrast to the 18% confidence level reported for the manual process ($p = .042$).

Discussion: The Paradox of Trust

The most significant finding of this research is the "Over-Trust" Paradox. Users perceived the CV prototype as more trustworthy than the manual system they had used for years. While high trust is generally desirable, the authors caution that it can lead to automation bias, where clinicians may stop critically evaluating the system's output.

The low trust in the status quo—where 45% of participants labeled the manual system as "deceptive"—suggests that the bar for new technology is remarkably low. Consequently, the ethical burden falls on CV developers to frame these systems as "decision-making aids" rather than replacements for human judgment. UI design must reinforce that the ultimate responsibility for resident care remains with the clinician.

Conclusion and Future Work

This study highlights that CV performance metrics like accuracy and mAP are insufficient when deploying technology in high-stakes environments like LTC. Future work will focus on long-term field studies to observe how trust evolves when users encounter real-world algorithmic errors. Additionally, researchers aim to identify specific UI cues, such as displaying segmentation masks, that can foster "appropriate" trust without encouraging dangerous over-reliance.

Summary updated

Abstract

In computer vision research, especially when novel applications of tools are developed, ethical implications around user perceptions of trust in the underlying technology should be considered and supported. Here, we describe an example of the incorporation of such considerations within the long-term care sector for tracking resident food and fluid intake. We highlight our recent user study conducted to develop a Goldilocks quality horizontal prototype designed to support trust cues in which perceived trust in our horizontal prototype was higher than the existing system in place. We discuss the importance and need for user engagement as part of ongoing computer vision-driven technology development and describe several important factors related to trust that are relevant to developing decision-making tools.
BibTeX
@misc{pfisterer2019food,
  author = {Kaylen J. Pfisterer and Jennifer Boger and Alexander Wong},
  title = {{Food for thought: Ethical considerations of user trust in computer vision}},
  year = {2019},
  eprint = {1905.12487},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.12487},
}
Figure