Unveiling Real-Life Effects of Online Photo Sharing

Van-Khoa Nguyen, Adrian Popescu, Jérôme Deshayes-ChossartWACV 2022

TL;DR

This paper introduces LERVUP, a novel method that predicts how sharing photos online might impact users in real-life situations like job applications or loan requests. By combining object detection with user-profile analysis and situation-specific ratings, LERVUP aims to increase user awareness about the potential consequences of their data sharing.

What Problem Does This Paper Address?

Online social networks (OSNs) offer free services in exchange for user data, which is often used in contexts beyond the user's initial intent. This data can influence crucial real-life decisions, such as job prospects or loan approvals, often in opaque ways. While previous work has explored how specific data points (like inferred demographics) affect outcomes, there's a need for a more comprehensive approach that considers the impact of visual content (photos) within specific real-life scenarios. The authors aim to bridge this gap by developing a system that can predict and communicate these potential impacts to users, thereby enhancing their awareness and control over their shared data.

Methods

The core of the LERVUP approach involves several key components:

  • Crowdsourced Situation Ratings: The authors collected ratings for various visual objects across four impactful situations: accommodation search (ACC), bank credit demand (BANK), IT job search (IT), and waiter job search (WAIT). This was done by asking participants to rate how an object's presence in a photo might influence their perception in each situation on a 7-point Likert scale.
  • Object Detection: They utilized pre-trained object detectors (both mobile-optimized and generic) to identify objects within user photos.
  • Focal Rating: To address the issue of less impactful objects potentially overwhelming more significant ones, they introduced a "focal rating" mechanism. This mechanism boosts the influence of objects with higher absolute ratings, inspired by attention mechanisms in deep learning.
  • Image-Level Descriptor: For each photo, they created a descriptor that aggregates information about detected objects, including the count of positively and negatively rated objects, and the average confidence of detections.
  • User-Level Descriptor: These image-level descriptors are then aggregated at the user level using a clustering approach (K-means) to create a compact feature vector representing the user's visual profile.
  • LERVUP Training: Finally, a random forest regression model is trained using these user-level descriptors to predict a user's profile rating for a given situation. This supervised learning approach aims to mimic human judgment in rating visual profiles.

Datasets

  • Object Detection Datasets: The authors merged three existing datasets: OpenImages, ImageNet, and COCO. This combined dataset contains 269 objects and over 137,000 images, with a limit of 1000 images per object to manage imbalance.
  • User Profile Dataset: They used 500 user profiles from the YFCC dataset, which contains images shared under Creative Commons licenses. This dataset was split into training (400 profiles) and validation (100 profiles) sets for evaluation.

Key Insights and Ideas

  • Context is Crucial: The study strongly supports the hypothesis that the perception and impact of shared visual content are highly context-dependent. An object that might be neutral in one situation could have a significant positive or negative effect in another.
  • Boosting High-Impact Objects: The "focal rating" mechanism, which amplifies the influence of objects with strong ratings, proved beneficial, especially for the LERVUP model. This suggests that users, like the model, tend to focus on more salient visual cues.
  • User-Level Representation Matters: Aggregating image-level information into a user-level descriptor, particularly through clustering, leads to a more robust representation of a user's visual profile than simply averaging object detections.
  • Real-Life Impact is Predictable: The research demonstrates that it's possible to predict the potential real-life consequences of sharing photos, offering a valuable tool for user awareness.
  • Mobile-First Detection is Challenging: While mobile-optimized detectors are necessary for on-device processing, they generally yield lower accuracy compared to more powerful generic models, highlighting a trade-off between performance and usability.

Key Results

  • LERVUPfr Outperforms Baselines: The LERVUP model, especially when incorporating the focal rating (LERVUPfr), achieved the best performance across most situations, with six out of eight correlations falling into the "strong" range (correlation > 0.5).
  • Situation-Specific Performance: Performance varied significantly by situation. The waiter job search (WAIT) showed the highest correlations (up to 0.68), likely due to the availability of relevant objects in the detection dataset. The bank loan (BANK) situation was the most challenging, with lower correlations, especially for mobile detectors.
  • Focal Rating Benefit: The focal rating mechanism improved LERVUP's performance by up to 15 points in the BANK situation with the MOBI detector, validating its utility.
  • Object Selection Impact: For baseline methods, object selection and individual detection thresholds had a larger impact than the focal rating.
  • Preliminary Experiment Success: A preliminary experiment showed that the proposed feedback mechanism, which explicitly links the image to the situation ("You are in situation Y. The app predicts that the image should not be shared because it automatically detected object X in the image. Object X is negatively perceived in situation Y?"), was significantly more effective (p < 0.001) than a generic feedback message.

Limitations and Future Work

  • Dataset Size and Coverage: The authors acknowledge that the user profile dataset is not large enough for robust train/validation/test splits. They also note that the object detection dataset could be enriched with more highly-rated objects, especially for challenging situations like BANK.
  • Detector Performance: The performance is heavily reliant on the quality of object detectors. The study suggests that using more powerful detectors could further improve results.
  • Bias in Data: Both human and technical biases are inherent in the approach. Crowdsourced ratings reflect average biases, and detector imperfections contribute to technical bias. Future work aims to address biases related to sensitive demographics.
  • Extensibility: The approach is designed to be extensible to other situations, data types (e.g., likes, text), and more advanced profile rating methods.
  • Explainability: While LERVUP aims for transparency, improving the explainability of learned models, particularly deep object detectors, remains a key challenge.
  • API Limitations: The mobile app implementation allows users to control local photos, but controlling already shared photos is limited by existing API restrictions.

Summary updated

BibTeX
@inproceedings{nguyen2022unveiling,
  author = {Van-Khoa Nguyen and Adrian Popescu and Jérôme Deshayes-Chossart},
  title = {{Unveiling Real-Life Effects of Online Photo Sharing}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure