Measuring Semantic Relatedness between Flickr Images: From a Social Tag Based View

Zheng Xu, Xiangfeng Luo, Yunhuai Liu, Lin Mei, Chuanping HuThe Scientific World JOURNAL 2014 · DOI

TL;DR

This paper proposes a novel method to measure the semantic relatedness between images using their social tags from platforms like Flickr. By leveraging tag co-occurrence, bipartite graph matching, and tag order, the approach effectively captures nuanced semantic relationships beyond simple visual similarity, leading to improved image clustering and search performance.

What Problem Does This Paper Address?

Traditional image similarity measures often rely on low-level visual features like texture or shape. While effective for some tasks, these methods struggle to capture the high-level semantic meaning that humans readily understand. The rise of social media platforms like Flickr has provided a wealth of user-generated tags that offer a rich source of semantic information. However, these tags can be noisy, redundant, and lack a formal ontology, making it challenging to directly use them for accurate semantic relatedness measurement. This paper aims to bridge this gap by developing a robust method to infer semantic relatedness between images based on their social tags, enabling more sophisticated multimedia applications.

Methods

The proposed method consists of three main steps:

  1. Tag Relatedness Computation: The paper explores four co-occurrence measures (Jaccard, Overlap, Dice, and PMI) to quantify the semantic relatedness between individual tag pairs. These measures leverage web search engine page counts to estimate how often tags appear together, reflecting their semantic association. PMI is ultimately chosen for its superior correlation with human judgment.
  2. Semantic Relatedness Integration: To measure the relatedness between two images, the tag-pair relatedness is framed as an assignment problem on a bipartite graph. This approach, termed maxRel, finds the optimal matching between tags of two images to maximize the overall semantic relatedness, effectively handling situations where tag sets have different sizes. This step aims to reduce noise and redundancy by considering the best possible pairings.
  3. Tag Order Revision: Recognizing that the order in which users provide tags can indicate their importance, the maxRel function is further refined. A "decline factor" is introduced, giving more weight to tags that appear earlier in a tag list. This heuristic acknowledges that users might prioritize more significant tags first. Additionally, a "tag pruning" schema is implemented to set the relatedness of identical tags to zero, preventing the method from collapsing into a simple similarity measure.

Datasets

The authors used a dataset of 1000 images from Flickr, organized into ten groups. These groups were further divided into two classes:

  • Class 1: Car, Phone, Flower, Dog, Boat (representing more general concepts)
  • Class 2: Louis Vuitton, Dior, Gucci, Cartier, Chanel (representing luxury brands, expected to have stronger semantic correlations)

This dataset was used for evaluating the proposed method's performance in clustering and image searching tasks.

Key Insights and Ideas

  • Social tags as a rich semantic source: The paper highlights the potential of user-generated tags on platforms like Flickr to capture semantic meaning, even for images that might appear visually dissimilar.
  • Beyond simple similarity: The distinction between semantic relatedness and semantic similarity is crucial. Relatedness can exist between dissimilar concepts (e.g., "doctor" and "hospital" through co-occurrence), which is what the proposed method aims to capture.
  • Tag order matters: The order in which users tag images is not arbitrary and can reflect the perceived importance of those tags, providing a valuable signal for semantic interpretation.
  • Bipartite matching for robust integration: Framing tag-pair relatedness as a bipartite graph assignment problem allows for a more comprehensive and robust integration of tag relationships, overcoming limitations of simpler aggregation methods.
  • Handling noisy social tags: The combination of co-occurrence measures, bipartite matching, and tag order weighting helps to mitigate the inherent noise and redundancy present in social tagging data.

Key Results

  • Improved Clustering: The proposed method, especially when incorporating tag order, significantly outperformed cosine-based clustering in terms of F-measure, Purity, and Entropy across both data set classes. The performance gain was more pronounced in Class 2, where semantic correlations were stronger.
  • Enhanced Image Searching: The method demonstrated superior performance in image searching tasks compared to a baseline Flickr search, achieving higher precision at cut-off points (P¹, P⁵, P¹⁰).
  • Robustness: The method proved to be robust across different data sets and demonstrated its ability to handle semantic relatedness searching effectively.

Limitations and Future Work

  • Limitations: The paper acknowledges that the effectiveness of the method relies on the semantic correlation within the data sets. While the proposed method performs well, it is noted that the performance difference between the proposed method and the cosine method was greater in Group 2 (luxury brands) due to stronger semantic correlations.
  • Future Work: The authors suggest that the proposed method can facilitate applications such as searching and recommendation. The paper also hints at extending the relatedness measures to videos with social tags. Further exploration into optimizing the tag order revision and exploring other semantic relatedness measures could be future directions.

Summary updated

Abstract

Relatedness measurement between multimedia such as images and videos plays an important role in computer vision, which is a base for many multimedia related applications including clustering, searching, recommendation, and annotation. Recently, with the explosion of social media, users can upload media data and annotate content with descriptive tags. In this paper, we aim at measuring the semantic relatedness of Flickr images. Firstly, four information theory based functions are used to measure the semantic relatedness of tags. Secondly, the integration of tags pair based on bipartite graph is proposed to remove the noise and redundancy. Thirdly, the order information of tags is added to measure the semantic relatedness, which emphasizes the tags with high positions. The data sets including 1000 images from Flickr are used to evaluate the proposed method. Two data mining tasks including clustering and searching are performed by the proposed method, which shows the effectiveness and robustness of the proposed method. Moreover, some applications such as searching and faceted exploration are introduced using the proposed method, which shows that the proposed method has broad prospects on web based tasks.
BibTeX
@article{xu2014measuring,
  author = {Zheng Xu and Xiangfeng Luo and Yunhuai Liu and Lin Mei and Chuanping Hu},
  title = {{Measuring Semantic Relatedness between Flickr Images: From a Social Tag Based View}},
  journal = {The Scientific World JOURNAL},
  year = {2014},
  doi = {10.1155/2014/758089},
}
Figure