UbiHearo: Bringing Scenario-Aware Sound Guidance to DHH Users with Mobile Agents

Fengmin Wu, Sicong Liu, Zimu Zhou, Chenren Xu, Wanbing Zhao, Peiwen Sun, Zhiwen YuarXiv 2026 · DOI · arXiv

TL;DR

UbiHearo is a mobile hearing assistant that moves beyond simple sound labeling to provide context-aware, actionable guidance for Deaf and Hard-of-Hearing (DHH) users. By combining energy-efficient "perception-thresholded" sensing with a staged neuro-symbolic reasoning framework, it delivers personalized safety and social cues directly on-device with minimal battery impact.

What Problem Does This Paper Address?

For DHH users, sound awareness isn't just about knowing what happened (e.g., "car horn"); it’s about knowing what it means and what to do. Existing mobile assistants often fail because they provide generic alerts that ignore context. A car horn heard while sitting in a living room is likely irrelevant background noise, but the same sound at a busy street crossing is a critical safety warning.

Current solutions suffer from three main gaps:

  1. Sensing: Always-on microphones drain batteries, but "duty-cycling" (turning the mic on and off) often misses brief, transient sounds like a shouted warning.
  2. Reasoning: Most apps map sounds to isolated labels. They lack the "common sense" to interpret how urgency changes across different scenarios.
  3. Action: Users are often left to infer the correct response under time pressure. There is a high demand for actionable guidance (e.g., "check the door" vs. "stay cautious") rather than just a notification.

Methods

UbiHearo employs a neuro-symbolic design that closes the sensing–reasoning–action loop through three novel mechanisms:

  • Perception-Thresholded Sensing: Inspired by the Weber–Fechner law, the system uses a lightweight "short-term listening" phase. It only activates full processing if a sound exceeds a "just-perceptible" energy threshold. Safety-critical sounds (like sirens) use a deterministic override to bypass complex reasoning and trigger instant alerts.
  • Staged Neuro-Symbolic Reasoning: Rather than feeding raw, noisy sensor data into a large model, UbiHearo uses Intermediate Representations (IRs).
  • Stage 1: A 1.5B-parameter model (Qwen2.5-Instruct) infers the physical context (e.g., "indoor/stationary" vs. "outdoor/mobile") from lightweight sensor cues like Wi-Fi density and step frequency.
  • Stage 2: These context labels act as structured prompts for an on-device 1.8B-parameter audio-language model (Qwen2-Audio), which generates specific guidance based on the sound and the scenario.
  • Bounded Human-in-the-Loop Personalization: To handle individual preferences (e.g., a student wanting to recognize a specific school bell), UbiHearo uses a lightweight output-layer adapter. This allows for on-device fine-tuning based on user feedback without "breaking" the model's general reasoning capabilities.

Datasets

The authors evaluated UbiHearo using:

  • Self-collected Physical Context Dataset: Over 1,000 clips containing accelerometer data, Wi-Fi AP counts, SSID semantics, and satellite SNR values to train the context inference model.
  • AudioSet: A standard large-scale dataset used for acoustic training and benchmarking.
  • User Studies: A survey of 100 DHH users to establish design goals, followed by a 21-day longitudinal field study with 5 participants and a "quick test" with 100 additional users.

Key Insights and Ideas

  • The "Perception Boundary" as a Filter: The realization that a mobile agent doesn't need to be "smarter" than a human ear allows for massive energy savings. If a sound is too quiet for a human to perceive, the system doesn't need to waste cycles processing it.
  • Structured IRs over Raw Streams: Multimodal LLMs often struggle with "hallucinations" when fed raw sensor streams. By converting sensors into symbolic "Intermediate Representations" first, the authors created a "scaffold" that makes the LLM's reasoning much more reliable and efficient.
  • Scenario Routing: By explicitly identifying the scenario first, the system can apply different "preference slots." This mimics human cognitive load management—prioritizing accuracy when stationary and responsiveness when mobile.

Key Results

  • Efficiency: UbiHearo consumes 391× less energy than heavy CNN baselines like YAMNet. It supports up to 12.5 hours of continuous operation on a standard smartphone.
  • Latency: The system achieves a mean end-to-end latency of ~1.5 seconds, making it practical for real-time safety alerts.
  • Guidance Quality: In blind pairwise comparisons against state-of-the-art models (like Qwen-Audio2 7B), UbiHearo achieved an 84% absolute preference score, significantly outperforming larger, non-staged models.
  • User Impact: In the field study, users reported high satisfaction (average score 4.6/5) and noted that the "customized mode" successfully learned personal sounds like microwave beeps or specific door chimes that generic datasets miss.

Limitations and Future Work

  • Sensor Robustness: The system's context awareness relies on GPS and Wi-Fi; weak signals in deep indoor environments can occasionally lead to incorrect scenario labeling.
  • Acoustic Confusion: Certain environmental sounds (e.g., loud wind or door slams) are still occasionally misclassified or confused with speech in high-noise environments.
  • Future Directions: The authors suggest exploring even more efficient distillation methods to reduce the 3.3B total parameter stack and incorporating richer low-power context signals (like light sensors or barometer data) to further refine scenario detection.

Summary updated

Abstract

For Deaf and Hard-of-Hearing (DHH) users, effective sound awareness goes beyond detecting \emph{what} sound occurs; it requires understanding \emph{what it means}, \emph{how urgent it is}, and \emph{what to do}. However, existing mobile assistants largely reduce acoustic scenes to isolated labels or generic alerts, failing to provide contextual guidance when the same sound implies different risks across scenarios. We present \sysname, a mobile hearing assistant agent that closes the sensing--reasoning--action loop for \textit{scenario-aware} and \textit{personalized} DHH assistance. \sysname adopts a \textit{neuro-symbolic} design that transforms continuous acoustic and physical context streams into structured intermediate representations (IRs) for controllable reasoning and guidance. It introduces three key mechanisms. \textit{First}, a perception-thresholded sensing mechanism enables energy-efficient always-on awareness by activating processing only for perceptually meaningful events while preserving safety-critical cues through deterministic overrides. \textit{Second}, a staged neuro-symbolic reasoning framework performs promptless scenario understanding by conditioning an on-device audio-language model on structured IRs instead of user-authored context descriptions. \textit{Third}, a bounded human-in-the-loop personalization mechanism adapts user-specific guidance through lightweight on-device decision-boundary updates while preserving population-aligned reasoning. We evaluate \sysname on smartphones and in-vehicle systems across diverse scenarios. It achieves favorable accuracy--latency--energy tradeoffs and personalized guidance.
BibTeX
@misc{wu2026ubihearo,
  author = {Fengmin Wu and Sicong Liu and Zimu Zhou and Chenren Xu and Wanbing Zhao and Peiwen Sun and Zhiwen Yu},
  title = {{UbiHearo: Bringing Scenario-Aware Sound Guidance to DHH Users with Mobile Agents}},
  year = {2026},
  eprint = {2611.20686},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20686},
}
Figure