Predicting Event Memorability Using Personalized Federated Learning

Sourasekhar Banerjee, Debaditya Roy, Vigneshwaran Subbaraju, Monowar BhuyanWACV 2025 · DOI

Introduction

Visual lifelogging, the practice of capturing one’s daily life through wearable cameras, generates a vast stream of images that can serve as powerful tools for reminiscence therapy and cognitive intervention. However, not every captured moment is an effective memory cue; many images are mundane, redundant, or poorly framed. The challenge lies in automatically predicting event memorability—the likelihood that a specific image will trigger a vivid recall of a past event.

Predicting memorability is inherently difficult due to two primary factors: privacy and subjectivity. Lifelog images are deeply personal, making traditional centralized machine learning (which requires data uploading) a significant privacy risk. Furthermore, memory is subjective; what is memorable to one person may be trivial to another, rendering "one-size-fits-all" global models ineffective.

The Challenge: Privacy vs. Personalization

Traditional machine learning paradigms present a difficult trade-off in the context of lifelogging. Centralized models benefit from large datasets but compromise user privacy and often fail to capture individual nuances. Conversely, "siloed" models—trained exclusively on a single user's data—protect privacy and offer personalization but suffer from a lack of sufficient training data to achieve high performance.

Comparison of centralized, siloed, and federated learning paradigms for lifelogging.

To bridge this gap, the authors introduce FedMEM, a framework utilizing Personalized Federated Learning (PFL). In this decentralized approach, sensitive images never leave the user's device; instead, only model weights (gradients) are shared with a central server to collaboratively improve the system while maintaining strict data privacy.

The FedMEM Framework

The FedMEM architecture employs a frozen ResNet50 backbone to extract visual features from images. To account for the multi-faceted nature of memory, the model utilizes a late-fusion approach, integrating visual data with 28 contextual features, including temporal data (time of day, duration of event) and visuo-semantic context (place, activity, presence of humans).

To address the subjectivity of memory, FedMEM introduces two novel clustering strategies that group users with similar memory patterns:

  1. Memorability-score Distribution-based Clustering (MSDC): An a priori method that groups users based on the statistical distribution of their self-reported memory scores.
  2. Model Similarity-based Clustering (MSC): A dynamic approach where the server groups users based on the similarity of their locally trained models. This method uses a specialized similarity measure to balance intra-cluster cohesion, inter-cluster distinction, and alignment with a global model.
The FedMEM+MSC dynamic clustering process, grouping users based on model similarity matrices.

Key Results and Insights

Evaluated on the R3 dataset—the only public dataset with lifelog images and memorability annotations—FedMEM demonstrated significant performance gains over traditional methods:

  • Superior Accuracy: FedMEM with Model Similarity Clustering (MSC) achieved an F1-score of 34.47%, more than doubling the performance of the centralized image-only baseline (15.00%).
  • The Contextual Boost: Integrating contextual features further improved the F1-score to 37.43%. Notably, the image-only federated model performed nearly as well as previous centralized models that required both images and context, highlighting the efficiency of the federated approach.
  • Personalization is Essential: The study found that even simple siloed models outperformed centralized ones, confirming that event memorability is too subjective for a single global average to be effective.
  • Robustness: The system maintained high performance even with low user participation (20%) or limited labeled data per user, making it viable for real-world applications where user engagement may vary.

Conclusion

FedMEM demonstrates that in the domain of subjective human experience, privacy and performance are not mutually exclusive. By leveraging personalized federated learning and intelligent clustering, the framework provides a scalable, private, and highly accurate solution for predicting event memorability. Future work will explore more complex temporal contexts and the application of these clustering techniques to other subjective computer vision tasks, such as aesthetic preference and emotional impact.

Summary updated

BibTeX
@inproceedings{banerjee2025predicting,
  author = {Sourasekhar Banerjee and Debaditya Roy and Vigneshwaran Subbaraju and Monowar Bhuyan},
  title = {{Predicting Event Memorability Using Personalized Federated Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00159},
}
Figure