Abstract
While users often attempt to conceal their social circles on Online Social Networks (OSNs), their diverse digital footprints—ranging from captions and hashtags to locations and images—can betray these hidden connections. This paper introduces a multimodal machine learning framework that integrates five distinct types of user-generated content to infer hidden friendships. Evaluating the framework on a large-scale Instagram dataset from Los Angeles and New York, the authors demonstrate that an adversary can predict relationships with over 90% accuracy (AUC > 0.9). These results suggest that current privacy defenses, which typically focus on a single data type, are insufficient against multi-faceted data leaks.
Introduction
In the era of Online Social Networks (OSNs), privacy is a primary concern. Users frequently hide their friendship lists to avoid surveillance or maintain professional boundaries. However, previous research has shown that these relationships can often be inferred through "inference attacks" using specific data types like shared locations or common hashtags.
Modern social media usage is inherently multimodal; a single post often contains a rich combination of visual, textual, and spatial data. This study investigates the compounding privacy risks when an adversary combines these disparate streams to reconstruct a user's social graph.
Methodology
The researchers developed a suite of binary classifiers using Random Forests to determine if a pair of users are friends. The framework leverages five data modalities:
- Textual Content: Captions are vectorized using TF-IDF, and eight pairwise distance metrics (e.g., Cosine, Euclidean) are calculated between user profiles.
- Hashtags: The system uses ten hand-engineered features, prioritizing "rare" hashtags through an information-theoretic approach (similar to the Adamic-Adar index) where low-entropy tags carry more weight.
- Visual Scenes: To avoid the complexities of facial recognition, the model uses a pre-trained ResNet18 (Places365) to extract scene categories (e.g., "café," "park") and compares the overlap in scene distributions between users.
- Location Data: User check-in behavior is modeled using
walk2friends, a method employing bipartite graphs and random walks to create behavioral embeddings. - Network Structure: The visible portions of the social graph are processed via
node2vecto predict missing edges.
The final Multimodal Fusion uses an ensemble approach, calculating a weighted average of the posterior probabilities from each individual classifier based on their validated confidence scores.
Experimental Results
The framework was tested on a massive dataset collected from Instagram, covering approximately 10,000 users in Los Angeles (LA) and 19,000 in New York (NY).
Performance of Individual Modalities
The study found that hashtags and text were the most potent standalone predictors. Hashtags, in particular, benefited from the "rarity equals intimacy" principle: sharing a niche, low-entropy hashtag is a much stronger signal of a real-world relationship than sharing common tags like #travel.
The Power of Fusion
The multimodal attack achieved an outstanding AUC of 0.925 in LA and 0.905 in NY, significantly outperforming any single-modality approach. Even the weakest modality—image scene analysis (AUC ~0.60-0.64)—contributed positively to the overall ensemble accuracy. Interestingly, the model proved highly resilient to data scarcity; even after randomly deleting 50% of user posts, the inference accuracy remained nearly stable.
Discussion and Conclusion
The findings highlight a significant "privacy gap." Users cannot easily protect themselves by simply posting less or obfuscating a single type of data (like location). Because different modalities complement each other, an adversary only needs a subset of available data (e.g., Text, Locations, and Network Edges) to achieve near-peak performance.
Future work should focus on temporal dynamics—such as the timing of co-occurring posts—and the development of multimodal obfuscation tools. As digital footprints become increasingly complex, privacy defenses must evolve to address the holistic picture of user behavior rather than isolated data points.
Summary updated