TL;DR
This paper tackles the challenge of efficiently summarizing vast amounts of cross-media and cross-genre information (like tweets and news articles) by proposing a novel "Multi-media Information Network" (MiNet) approach. By inferring implicit similarities across different data types and genres, their method significantly improves the ranking of informative content, saving users time and effort.
What Problem Does This Paper Address?
In today's digital landscape, information about a specific event or topic is scattered across various sources like news websites, social media (tweets), and images. This makes it incredibly difficult and time-consuming for users to sift through the noise and find the most relevant and informative content. Existing search engines and summarization techniques often struggle with this "cross-media" and "cross-genre" challenge, either focusing on a single data type or lacking robust methods to connect information across different formats and styles. The paper aims to bridge this gap by creating a unified representation and ranking system for this diverse information landscape.
Methods
The core of the proposed solution lies in constructing a Multi-media Information Network (MiNet). This involves:
- Information Extraction: Extracting salient facts and concepts from various sources, including:
- Texts: Using advanced Information Extraction (IE) systems to identify entities, relations, and events (e.g., from news articles). They also leverage Semantic Role Labeling (SRL) to capture more granular event details.
- Images: Employing a Probabilistic Logical Tree (PLT) approach to extract hierarchical concepts from images, integrating logical and statistical inferences.
- Web Documents: Retrieving and processing web documents linked from tweets to enrich the information pool.
- MiNet Construction: Building a heterogeneous network where nodes represent extracted concepts, entities, and events, and edges represent relationships between them. This network integrates information from tweets, web documents, and images.
- Cross-media and Cross-genre Linking: Developing methods to infer similarity between different data modalities:
- Text-Text Similarity: Using cosine similarity on frequency-weighted vectors of concepts and relations.
- Image-Image Similarity: Employing Hierarchical Gaussianization and Gaussian Mixture Models on HOG features to represent images and compute cosine similarity.
- Cross-media Linking: A novel approach using transfer learning to learn a cross-domain similarity function between image and text representations.
- MiNet-Based Information Ranking (EN-Tri-HITS): An extension of the Tri-HITS algorithm that uses a random walk on the constructed MiNet. This algorithm iteratively propagates ranking scores across tweets, sentences from web documents, and images, allowing for mutual reinforcement and refinement of initial rankings.
Datasets
The paper describes building their own dataset for the "Hurricane Irene" scenario. This involved:
- Crawling 3471 tweets.
- Extracting key phrases from these tweets to use as queries for Bing Image Search, retrieving the top 10 images for each query.
- Extracting image features (GIST and HOG) for meta-information training and similarity calculation.
- Creating ground truth labels based on human assessment of informativeness on a 5-star Likert scale.
- The authors note that currently, no publicly available information ranking benchmark datasets exist for this type of cross-media, cross-genre task.
Key Insights and Ideas
- The Power of Heterogeneous Networks: Integrating information from multiple sources (tweets, web docs, images) and genres significantly boosts ranking performance compared to single-modality approaches.
- Web Documents as Bridges: Formal genre web documents act as crucial bridges, helping to infer informative content from less structured sources like tweets.
- Cross-media Inference is Key: Simply fusing information isn't enough; inferring implicit similarities across media types (text to image, image to text) is essential for effective ranking.
- MiNets as a Unified Representation: The MiNet concept provides a principled way to represent and reason about complex relationships within and across diverse information sources.
Key Results
- Image Ranking: Methods integrating web documents with images outperformed a baseline image ranking method (0.421) with an NDCG@5 score of 0.583 when using both text and web documents. Integrating tweets alone did not significantly improve image ranking.
- Tweet Ranking: Methods that integrated heterogeneous networks (tweets with web documents) outperformed the baseline TextRank. Specifically, aligning tweets with web documents improved ranking quality significantly, with an NDCG@5 score of 0.809 when using tweets, web documents, and images.
- The proposed EN-Tri-HITS algorithm demonstrated significant improvements in ranking quality across experiments.
Limitations and Future Work
- Ambiguity and Spam: The system can be affected by ambiguous terms or spam tweets that are not topically related to the event, potentially leading to irrelevant content being ranked highly.
- Deep Semantic Analysis: While the system extracts concepts, there's a need for more extensive semantic analysis, especially for images, to better identify synonymous terms and richer relationships.
- Future Work: The authors suggest enhancing concept extraction quality by exploiting cross-media inference beyond simple fusion. They also aim to address the issues of ambiguity and spam in future work.
Summary updated