HandsOnAid: Building and Evaluating an Offline First-Aid Assistant for Emergency Response Scenarios

Rezwan Islam Salvi, Saiyma Sittul Muna, Mushfiqur Rahman MushfiquearXiv 2026 · arXiv

TL;DR

HandsOnAid explores the feasibility of deploying a 2-billion parameter language model (Gemma 2B) directly on a smartphone to provide offline first-aid guidance during emergencies. While fine-tuning and retrieval-augmented generation significantly improve performance, the study highlights a critical "alignment gap" between general training data and strict medical protocols, rendering the current system a research prototype rather than a deployable tool.

What Problem Does This Paper Address?

In a disaster scenario—an earthquake, a remote hike, or a mid-ocean emergency—cell service is often the first thing to go. While first-aid apps exist, they are often static libraries. Large Language Models (LLMs) offer the promise of interactive, situation-specific guidance, but they typically require massive server-side compute.

The authors tackle two main challenges:

  1. Efficiency: Can a model small enough to run on a mid-range smartphone (offline) provide useful medical advice?
  2. Safety: In first aid, "almost right" can be fatal. The paper investigates how to evaluate and improve the safety of these small models using strict medical guidelines (ANZCOR).

Methods

The researchers built HandsOnAid by starting with Google’s Gemma 2B model. They employed several techniques to optimize it for mobile and medical use:

  • Quantization & Fine-tuning: They used 4-bit quantization to shrink the model and QLoRA (Quantized Low-Rank Adaptation) to fine-tune it on first-aid data without needing massive hardware.
  • Retrieval-Augmented Generation (RAG): They tested a configuration that uses BM25 (a keyword-based search) to find a relevant example from the training set and show it to the model before it answers.
  • Safety Guardrails: They implemented a "safety gate" (a second pass where the model checks its own answer) and a "length floor" to prevent dangerously brief responses.
  • LLM-as-a-Judge: To evaluate the results, they used four "frontier" models (like GPT-5.6 and Claude Opus 5) to blindly score answers based on a strict 0–5 rubric, supplemented by a medical professional's review.
Overview of the study: FirstAidQA dataset, Fine-tuning, 7 configurations, 41-question exam, Rubric and 4 LLM judges.

Datasets

  • Training: FirstAidQA, a corpus of 5,550 synthetic question-answer pairs.
  • Evaluation: A newly created 41-question Evaluation Bank. These questions were designed to be similar to the training data but were not copied from it. Crucially, the reference answers were aligned with ANZCOR (Australian and New Zealand Committee on Resuscitation) guidelines and reviewed by a medical professional.

Key Insights and Ideas

  • The Data Alignment Gap: The most striking finding was that even the best-performing models scored poorly (averaging ~2.5/5). A keyword audit revealed that the training data simply didn't contain the specific, high-stakes details required by the evaluation rubric (e.g., the exact 30:2 CPR compression-to-breath ratio).
  • Quantization is "Free": There was negligible difference in accuracy between the 4-bit and 8-bit fine-tuned models. For medical tasks at this scale, the memory savings of 4-bit quantization don't seem to come at a high cost to safety.
  • Retrieval is a Double-Edged Sword: While RAG provided the highest overall scores, it only helped when the retrieved training example was high-quality. If the training set had a gap, the retrieval could actually lead the model astray.
  • Small Models Struggle with Self-Correction: The "safety gate" (where the 2B model checks itself) was largely ineffective at improving scores, often just defaulting to a generic "stay still" fallback message.

Key Results

The study compared seven configurations, as detailed in the table below:

The seven configurations tested in the study.
  • Performance Boost: Fine-tuning improved the mean score by +1.08 points over the base model.
  • Top Performer: The fine-tuned 4-bit model with retrieval (Config F) achieved the highest score of 2.51/5.
  • Safety: Fine-tuning reduced the share of "dangerous or wrong" (0–1) ratings from 64.6% in the base model to 25.0%.
  • On-Device Speed: Running on a Google Pixel 7 Pro (6 CPU threads), the model produced its first text in 6.86 seconds and a full answer in roughly 15 seconds.
Share of judge scores in each band (0–1, 2, 3–4, 5) for the seven configurations.

Limitations and Future Work

The authors are transparent: HandsOnAid is not ready for the field. A score of 2.51/5 means the model still misses critical steps or provides incomplete advice too often.

Future directions include: * Better Training Data: Aligning the training corpus more strictly with established medical protocols like ANZCOR or AHA. * Multi-turn Interaction: Allowing the model to ask clarifying questions (e.g., "Is the patient breathing?") before giving advice. * Human Validation: Expanding the evaluation to include a larger panel of medical professionals across different regions and languages.

Summary updated

Abstract

Emergencies can occur where there is no phone signal and no ambulance nearby. A small language model on a phone can be helpful in such situations, but its advice needs to be safe and accurate. We built HandsOnAid, a 4-bit Gemma 2B model fine-tuned with QLoRA on FirstAidQA dataset, a corpus of 5,550 first-aid question-answer pairs. We compared seven configurations, including the base, 4-bit and 8-bit fine-tuned models, and versions with a length floor, safety gate, or retrieval. We used a 41-pair question-answer bank with medically reviewed references that followed ANZCOR guidelines, and a strict 0-5 score rubric, with twelve safety overrides. Four frontier LLM judges (GPT-5.6 Sol, Claude Opus 5, DeepSeek V4 Pro, GLM-5.3) blindly scored every answer, with 45 hidden controls used to check judge reliability. Fine-tuning improved the mean score by +1.08 over the base model to 2.34. The best configuration scored 2.51 out of 5 which was the fine-tuned 4-bit model with retrieval. A medical professional also conducted a blind evaluation of 40 answers across the base, the 4-bit fine-tuned, and the retrieval configurations, and the scores were similar to the LLM judges' average scores. A keyword audit found limited coverage of several details of ANZCOR guidelines in the training corpus, that are required by the rubric, which suggests that missing information in the training data could be a cause of the low scores. On a smartphone device (Pixel 7 Pro) with six CPU threads and no network connection, the 4-bit fine-tuned model produced its first displayable text in 6.86 seconds and a complete answer in 14.88 seconds on average over ten questions.
BibTeX
@misc{salvi2026handsonaid,
  author = {Rezwan Islam Salvi and Saiyma Sittul Muna and Mushfiqur Rahman Mushfique},
  title = {{HandsOnAid: Building and Evaluating an Offline First-Aid Assistant for Emergency Response Scenarios}},
  year = {2026},
  eprint = {2611.14059},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14059},
}
Figure