PersoNo: Personalised Notification Urgency Classifier in Mixed Reality

Jingyao Zheng, Haodi Weng, Xian Wang, Chengbin Cui, Sven Mayer, Chi-lok Tai, Lik-Hang Lee

TL;DR

This paper introduces PersoNo, a personalized notification urgency classifier for Mixed Reality (MR) that intelligently filters notifications based on user preferences and context. By analyzing user behavior in MR, they found that activity context is as crucial as content and sender, leading to an 81.5% accurate system that significantly reduces false negatives.

What Problem Does This Paper Address?

Mixed Reality (MR) is increasingly integrated into daily life, offering enhanced capabilities. However, users are bombarded with notifications, disrupting their immersive experience. While mobile notification management has been studied extensively, MR presents unique challenges: users can't easily "put away" their MR devices without losing augmented capabilities. This paper addresses the need for intelligent MR notification classifiers that can filter alerts effectively, reducing distractions while ensuring important information is not missed. They aim to understand how users respond to notifications in MR and develop a personalized system to manage them.

Methods

The authors developed PersoNo, a personalized notification urgency classifier, through a three-stage research approach:

  • Data Collection: They created a novel MR notification dataset by conducting a user study (N=18) where participants wore MR headsets and performed everyday tasks while receiving notifications. This dataset includes both self-labeled urgency ratings and objective interaction-based data (response time).
  • Contextual Analysis: Through thematic analysis of user behavior patterns, they identified key factors influencing notification urgency in MR.
  • Classifier Development: They leveraged Large Language Models (LLMs) and a multi-agent approach to build PersoNo. The system uses an "AI analyzer" to create user profiles from past notification interaction history, which "AI raters" then use to evaluate incoming notifications. They explored two LLM prompting techniques: zero-shot learning (M1) and a multi-agent method (M2) for personalized classification, comparing them against baseline general classifiers.

Datasets

  • MR Notification Dataset: A newly introduced dataset collected from 18 participants during MR activities. It contains both:
  • Self-labeled data: 90 notifications per participant, where users rated urgency.
  • Interaction-based data: 108 notifications per participant, classified as urgent or non-urgent based on response time (within 30 seconds for urgent).
  • Mobile Text Dataset (mobile_train.txt): Used as a source for message content, with sender placeholders assigned to emulate real-world scenarios.

Key Insights and Ideas

  • Activity Context is King (in MR): Unlike mobile notifications where content and sender are often paramount, in MR, the user's current activity is equally, if not more, important for determining notification urgency. This is a significant departure from previous mobile-centric research.
  • Personalization Outperforms Generalization: Personalized classifiers (M1 and M2) significantly outperformed general classifiers (Base) in accuracy and, crucially, in reducing false negative rates.
  • Self-Reported Data is Viable: The study found that self-reported urgency ratings can serve as a viable alternative to time-consuming interaction-based data collection for training personalized classifiers, a finding that contradicts some prior work.
  • LLMs for Personalized MR Notifications: Large Language Models, particularly with a multi-agent approach and in-context learning, are well-suited for building personalized MR notification classifiers, even with limited user data.
  • Human-Centered AI: The use of LLMs with Chain-of-Thought prompting enhances transparency, allowing users to understand why a notification is classified as urgent, fostering trust and control.

Key Results

  • Accuracy: PersoNo (M2 with D2 dataset) achieved 81.5% accuracy.
  • False Negative Rate (FNR): PersoNo significantly reduced FNR to 0.381, meaning fewer urgent notifications were missed compared to baseline models.
  • AUROC: The best performing model achieved an AUROC of 0.786.
  • Contextual Impact: Incorporating "Activity" context (D2 dataset) significantly improved performance (lower FNR, higher accuracy) compared to models using only "Sender" and "Content" (D1 dataset).
  • Self-Report vs. Interaction: Models trained on self-reported data (SR) showed comparable performance to those trained on interaction-based data (D1/D2), with no statistically significant differences in accuracy or FNR.

Limitations and Future Work

  • Controlled Environment: The study was conducted in a controlled lab setting. Future work should involve "in-the-wild" deployments to assess PersoNo's performance under more authentic and evolving social conditions.
  • Limited Contextual Factors: While activity context was crucial, future research could explore other factors like time of day, location, immersion level, and the interplay between virtual and physical tasks.
  • Binary Classification: The current system uses binary urgency classification. Extending this to multi-level urgency categories could unlock more nuanced notification placement strategies.
  • User Adaptation: While PersoNo adapts to user behavior, future work could explore how the system can continuously adapt to changing user patterns over time.
  • Generalizability of Self-Report: While self-reported data proved effective here, further investigation into its reliability across different types of HCI tasks and user populations is warranted.

Summary updated

Abstract

Mixed Reality (MR) is increasingly integrated into daily life, providing enhanced capabilities across various domains. However, users face growing notification streams that disrupt their immersive experience. We present PersoNo, a personalised notification urgency classifier for MR that intelligently classifies notifications based on individual user preferences. Through a user study (N=18), we created the first MR notification dataset containing both self-labelled and interaction-based data across activities with varying cognitive demands. Our thematic analysis revealed that, unlike in mobiles, the activity context is equally important as the content and the sender in determining notification urgency in MR. Leveraging these insights, we developed PersoNo using large language models that analyse users replying behaviour patterns. Our multi-agent approach achieved 81.5% accuracy and significantly reduced false negative rates (0.381) compared to baseline models. PersoNo has the potential not only to reduce unnecessary interruptions but also to offer users understanding and control of the system, adhering to Human-Centered Artificial Intelligence design principles.
BibTeX
@inproceedings{zheng2025persono,
  author = {Jingyao Zheng and Haodi Weng and Xian Wang and Chengbin Cui and Sven Mayer and Chi-lok Tai and Lik-Hang Lee},
  title = {{PersoNo: Personalised Notification Urgency Classifier in Mixed Reality}},
  booktitle = {Ismar},
  year = {2025},
}
Figure