TL;DR
UbiHearo is a mobile hearing assistant that moves beyond simple sound labeling to provide context-aware, actionable guidance for Deaf and Hard-of-Hearing (DHH) users. By combining energy-efficient "perception-thresholded" sensing with a staged neuro-symbolic reasoning framework, it delivers personalized safety and social cues directly on-device with minimal battery impact.
What Problem Does This Paper Address?
For DHH users, sound awareness isn't just about knowing what happened (e.g., "car horn"); it’s about knowing what it means and what to do. Existing mobile assistants often fail because they provide generic alerts that ignore context. A car horn heard while sitting in a living room is likely irrelevant background noise, but the same sound at a busy street crossing is a critical safety warning.
Current solutions suffer from three main gaps:
- Sensing: Always-on microphones drain batteries, but "duty-cycling" (turning the mic on and off) often misses brief, transient sounds like a shouted warning.
- Reasoning: Most apps map sounds to isolated labels. They lack the "common sense" to interpret how urgency changes across different scenarios.
- Action: Users are often left to infer the correct response under time pressure. There is a high demand for actionable guidance (e.g., "check the door" vs. "stay cautious") rather than just a notification.
Methods
UbiHearo employs a neuro-symbolic design that closes the sensing–reasoning–action loop through three novel mechanisms:
- Perception-Thresholded Sensing: Inspired by the Weber–Fechner law, the system uses a lightweight "short-term listening" phase. It only activates full processing if a sound exceeds a "just-perceptible" energy threshold. Safety-critical sounds (like sirens) use a deterministic override to bypass complex reasoning and trigger instant alerts.
- Staged Neuro-Symbolic Reasoning: Rather than feeding raw, noisy sensor data into a large model, UbiHearo uses Intermediate Representations (IRs).
- Stage 1: A 1.5B-parameter model (Qwen2.5-Instruct) infers the physical context (e.g., "indoor/stationary" vs. "outdoor/mobile") from lightweight sensor cues like Wi-Fi density and step frequency.
- Stage 2: These context labels act as structured prompts for an on-device 1.8B-parameter audio-language model (Qwen2-Audio), which generates specific guidance based on the sound and the scenario.
- Bounded Human-in-the-Loop Personalization: To handle individual preferences (e.g., a student wanting to recognize a specific school bell), UbiHearo uses a lightweight output-layer adapter. This allows for on-device fine-tuning based on user feedback without "breaking" the model's general reasoning capabilities.
Datasets
The authors evaluated UbiHearo using:
- Self-collected Physical Context Dataset: Over 1,000 clips containing accelerometer data, Wi-Fi AP counts, SSID semantics, and satellite SNR values to train the context inference model.
- AudioSet: A standard large-scale dataset used for acoustic training and benchmarking.
- User Studies: A survey of 100 DHH users to establish design goals, followed by a 21-day longitudinal field study with 5 participants and a "quick test" with 100 additional users.
Key Insights and Ideas
- The "Perception Boundary" as a Filter: The realization that a mobile agent doesn't need to be "smarter" than a human ear allows for massive energy savings. If a sound is too quiet for a human to perceive, the system doesn't need to waste cycles processing it.
- Structured IRs over Raw Streams: Multimodal LLMs often struggle with "hallucinations" when fed raw sensor streams. By converting sensors into symbolic "Intermediate Representations" first, the authors created a "scaffold" that makes the LLM's reasoning much more reliable and efficient.
- Scenario Routing: By explicitly identifying the scenario first, the system can apply different "preference slots." This mimics human cognitive load management—prioritizing accuracy when stationary and responsiveness when mobile.
Key Results
- Efficiency: UbiHearo consumes 391× less energy than heavy CNN baselines like YAMNet. It supports up to 12.5 hours of continuous operation on a standard smartphone.
- Latency: The system achieves a mean end-to-end latency of ~1.5 seconds, making it practical for real-time safety alerts.
- Guidance Quality: In blind pairwise comparisons against state-of-the-art models (like Qwen-Audio2 7B), UbiHearo achieved an 84% absolute preference score, significantly outperforming larger, non-staged models.
- User Impact: In the field study, users reported high satisfaction (average score 4.6/5) and noted that the "customized mode" successfully learned personal sounds like microwave beeps or specific door chimes that generic datasets miss.
Limitations and Future Work
- Sensor Robustness: The system's context awareness relies on GPS and Wi-Fi; weak signals in deep indoor environments can occasionally lead to incorrect scenario labeling.
- Acoustic Confusion: Certain environmental sounds (e.g., loud wind or door slams) are still occasionally misclassified or confused with speech in high-noise environments.
- Future Directions: The authors suggest exploring even more efficient distillation methods to reduce the 3.3B total parameter stack and incorporating richer low-power context signals (like light sensors or barometer data) to further refine scenario detection.
Summary updated