Online social image ranking in diversified preferences

Xuezhuan Zhao, Lishen Pei, Tao Li, Zheng ZhangEURASIP 2020 · DOI

Abstract

Due to the prevalence of social media services, effective and efficient online image retrieval is in urgent need to satisfy diversified requirements of Web users. Previous studies mainly focus on bridging the semantic gap by well-established content modeling with semantic information and social tagging information, but they are not flexible in aggregating the diversified expectations of online users. In this paper, we present OSIR, a solution framework to facilitate diversified preference styles in online social media image searching by textual query inputs. First, we propose an efficient Online Multiple Kernel Ranking (OMKR) model constructed on multiple query dimensions and complimentary feature channels, trained by minimizing the triplet loss on hard negative samples. Second, we construct random walk-based preference modeling by domain-specific similarity calculation on heterogeneous social attributes. Finally, we propose an effective and efficient position-sensitive rank aggregation approach to aggregate multiple ranking results based on user preference specification. Extensive experiments on two social media datasets demonstrate the advantages of our approach in both retrieval performance and user experience.

Introduction: Bridging the Intention Gap

Traditional image retrieval systems face two major hurdles: the semantic gap (mapping pixels to concepts) and the intention gap. The latter occurs because different users entering the same query—such as "sunflower"—may have vastly different intents; one might seek wild landscapes while another wants botanical close-ups. Most existing models capture an "average" community preference, returning identical results to everyone. Furthermore, many rely on slow, inflexible intermediate semantic annotations.

The OSIR (Online Social Image Ranking) framework addresses these challenges by creating a system that is both online-learnable—capable of processing massive social media streams without full database reloads—and preference-aware, leveraging social metadata to diversify results.

The OSIR Framework

The OSIR framework integrates three core components to deliver semantically accurate and personalized search results.

The OSIR framework overview, illustrating the integration of Online Multiple Kernel Ranking, preference modeling across social attributes, and final rank aggregation.

1. Online Multiple Kernel Ranking (OMKR)

OMKR learns to project images and textual queries into a shared semantic space using a combination of visual kernels (e.g., color histograms, deep CNN layers). Weights are updated using a fusion of Perceptron and Hedge algorithms. To maintain efficiency, the system employs Hard Negative Mining via Kernelized Locality Sensitivity Hashing (KLSH), focusing training on difficult examples that most significantly improve ranking accuracy.

2. Random Walk-based Preference Modeling

Social attributes—such as upload time, GPS location, user ID, and group membership—are treated as a graph. By performing random walks on these heterogeneous attributes, the model generates multiple "preference-style" ranking lists. For example, one list might prioritize images from a specific location, while another favors a user's social circle.

3. Position-Sensitive Rank Aggregation

Merging multiple ranked lists is a complex task. OSIR utilizes a weighted, position-sensitive approach. Unlike standard methods that treat every position equally, this algorithm uses a logarithmic penalty to ensure that "top" results (those most visible to users) are prioritized and protected during the merging process.

Experimental Evaluation

The authors evaluated OSIR on two large-scale real-world datasets:

  • NUS-WIDE: 269,648 images with 81 ground-truth concept tags and social metadata.
  • Flickr Dataset: 3.5 million images with 150 common queries and diverse social attributes.

Key Results

  • Superior Accuracy: OSIR consistently outperformed state-of-the-art methods like PAMIR. On NUS-WIDE, the deep version of OSIR achieved a Mean Average Precision (MAP) of 44.4%, compared to 32.6% for the baseline.
  • Subjective Satisfaction: Human evaluators gave the aggregated ranking significantly higher NDCG@50 scores, confirming better alignment with actual human preferences.
  • Efficiency: The OMKR model converges quickly, requiring fewer than 30,000 training triplets to achieve high performance, making it viable for real-time deployment.

Discussion and Conclusion

The OSIR framework demonstrates that social metadata is a powerful signal for capturing user intent, acting as the "mixer" in a "cocktail" of personalized results where semantic ranking is the base. By protecting the "Top-K" results through position-sensitive aggregation, the system ensures relevance where it matters most.

While the system is highly effective, its performance remains dependent on the initial semantic ranking accuracy. Future work will explore incorporating real-time clickstream data and more complex social network structures to further refine the personalization of social image search.

Summary updated

Abstract

Abstract Due to the prevalence of social media service, effective and efficient online image retrieval is in urgent need to satisfy diversified requirements of Web users. Previous studies are mainly focusing on bridging the semantic gap by well-established content modeling with semantic information and social tagging information, but they are not flexible in aggregating the diversified expectations of the online users. In this paper, we present OSIR, a solution framework to facilitate the diversified preference styles in online social media image searching by textual query inputs. First, we propose an efficient Online Multiple Kernel Ranking (OMKR) model which is constructed on multiple query dimensions and complimentary feature channels, and trained by minimizing the triplet loss on hard negative samples. By optimizing the ranking performance with multi-dimensional queries, the semantic consistency between the image ranking and textual query input is directly maximized without relying on the intermediate semantic annotation procedure. Second, we construct random walk-based preference modeling by domain-specific similarity calculation on heterogeneous social attributes. By re-ranking the rank output of OMKR based on each preference ranking model, we obtain a set of ranking lists encoding different potential aspects of user preference. Last, we propose an effective and efficient position-sensitive rank aggregation approach to aggregate multiple ranking results based on the user preference specification. Extensive experiment on two social media datasets demonstrates the advantages of our approach in both retrieval performance and user experience.
BibTeX
@inproceedings{zhao2020online,
  author = {Xuezhuan Zhao and Lishen Pei and Tao Li and Zheng Zhang},
  title = {{Online social image ranking in diversified preferences}},
  booktitle = {EURASIP},
  year = {2020},
  doi = {10.1186/s13640-020-00540-4},
}
Figure