Everything About You: A Multimodal Approach towards Friendship Inference in Online Social Networks

Tahleen A. Rahman, Mario Fritz, Michael Backes, Yang ZhangarXiv 2020 · OpenReview

Abstract

While users often attempt to conceal their social circles on Online Social Networks (OSNs), their diverse digital footprints—ranging from captions and hashtags to locations and images—can betray these hidden connections. This paper introduces a multimodal machine learning framework that integrates five distinct types of user-generated content to infer hidden friendships. Evaluating the framework on a large-scale Instagram dataset from Los Angeles and New York, the authors demonstrate that an adversary can predict relationships with over 90% accuracy (AUC > 0.9). These results suggest that current privacy defenses, which typically focus on a single data type, are insufficient against multi-faceted data leaks.

Introduction

In the era of Online Social Networks (OSNs), privacy is a primary concern. Users frequently hide their friendship lists to avoid surveillance or maintain professional boundaries. However, previous research has shown that these relationships can often be inferred through "inference attacks" using specific data types like shared locations or common hashtags.

Modern social media usage is inherently multimodal; a single post often contains a rich combination of visual, textual, and spatial data. This study investigates the compounding privacy risks when an adversary combines these disparate streams to reconstruct a user's social graph.

Figure 1: An example of the multimodal data available in a typical Instagram post, including image content, location metadata, captions, and hashtags.

Methodology

The researchers developed a suite of binary classifiers using Random Forests to determine if a pair of users are friends. The framework leverages five data modalities:

  1. Textual Content: Captions are vectorized using TF-IDF, and eight pairwise distance metrics (e.g., Cosine, Euclidean) are calculated between user profiles.
  2. Hashtags: The system uses ten hand-engineered features, prioritizing "rare" hashtags through an information-theoretic approach (similar to the Adamic-Adar index) where low-entropy tags carry more weight.
  3. Visual Scenes: To avoid the complexities of facial recognition, the model uses a pre-trained ResNet18 (Places365) to extract scene categories (e.g., "café," "park") and compares the overlap in scene distributions between users.
  4. Location Data: User check-in behavior is modeled using walk2friends, a method employing bipartite graphs and random walks to create behavioral embeddings.
  5. Network Structure: The visible portions of the social graph are processed via node2vec to predict missing edges.

The final Multimodal Fusion uses an ensemble approach, calculating a weighted average of the posterior probabilities from each individual classifier based on their validated confidence scores.

Experimental Results

The framework was tested on a massive dataset collected from Instagram, covering approximately 10,000 users in Los Angeles (LA) and 19,000 in New York (NY).

Performance of Individual Modalities

The study found that hashtags and text were the most potent standalone predictors. Hashtags, in particular, benefited from the "rarity equals intimacy" principle: sharing a niche, low-entropy hashtag is a much stronger signal of a real-world relationship than sharing common tags like #travel.

Figure 5: Comparison of AUC scores for various hashtag-based features in New York and Los Angeles, highlighting the effectiveness of hand-engineered entropy-based metrics.

The Power of Fusion

The multimodal attack achieved an outstanding AUC of 0.925 in LA and 0.905 in NY, significantly outperforming any single-modality approach. Even the weakest modality—image scene analysis (AUC ~0.60-0.64)—contributed positively to the overall ensemble accuracy. Interestingly, the model proved highly resilient to data scarcity; even after randomly deleting 50% of user posts, the inference accuracy remained nearly stable.

Discussion and Conclusion

The findings highlight a significant "privacy gap." Users cannot easily protect themselves by simply posting less or obfuscating a single type of data (like location). Because different modalities complement each other, an adversary only needs a subset of available data (e.g., Text, Locations, and Network Edges) to achieve near-peak performance.

Future work should focus on temporal dynamics—such as the timing of co-occurring posts—and the development of multimodal obfuscation tools. As digital footprints become increasingly complex, privacy defenses must evolve to address the holistic picture of user behavior rather than isolated data points.

Summary updated

Abstract

Most previous works in privacy of Online Social Networks (OSN) focus on a restricted scenario of using one type of information to infer another type of information or using only static profile data such as username, profile picture or home location. However the multimedia footprints of users has become extremely diverse nowadays. In reality, an adversary would exploit all types of information obtainable over time, to achieve its goal. In this paper, we analyse OSN privacy by jointly exploiting longterm multimodal information. We focus in particular on inference of social relationships. We consider five popular components of posts shared by users, namely images, hashtags, captions, geo-locations and published friendships. Large scale evaluation on a real-world OSN dataset shows that while our monomodal attacks achieve strong predictions, our multimodal attack leads to a stronger performance with AUC (area under the ROC curve) above 0.9. Our results highlight the need for multimodal obfuscation approaches towards protecting privacy in an era where multimedia footprints of users get increasingly diverse.
BibTeX
@misc{rahman2020everything,
  author = {Tahleen A. Rahman and Mario Fritz and Michael Backes and Yang Zhang},
  title = {{Everything About You: A Multimodal Approach towards Friendship Inference in Online Social Networks}},
  year = {2020},
  eprint = {2003.00996},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.00996},
}
Figure