Islander: A Real-Time News Monitoring and Analysis System

Chao-Wei Huang, Kai-Chou Yang, Zi-Yuan Chen, Hao-Chien Cheng, Po-Yu Wu, Yu-Yang Huang, Chung-Kai Hsieh, Geng-Zhi Wildsky Fann, Ting-Yin Cheng, Ethan Tu, Yun-Nung ChenarXiv 2022 · OpenReview

TL;DR

The "Islander" system tackles the overwhelming deluge of online news by providing a real-time monitoring and analysis platform. It goes beyond simple aggregation to assess news quality using metrics like objectivity, neutrality, and non-incitement, offering users a way to navigate the information landscape more effectively and identify potentially manipulated content.

What Problem Does This Paper Address?

In today's digital age, news consumption has shifted overwhelmingly online, with thousands of articles published daily from numerous sources. This sheer volume makes it challenging for readers to acquire information efficiently. Compounding this issue, many online media outlets prioritize engagement over accuracy, employing sensationalized headlines and content to attract clicks. This can lead to subjective, biased, or even inaccurate reporting, making it difficult for users to discern reliable information. Furthermore, the spread of misinformation and the potential for public opinion manipulation through coordinated online behaviors are significant concerns. Islander aims to address these challenges by providing a system that not only monitors news in real-time but also analyzes its quality and identifies potentially manipulated public discourse.

Methods

Islander employs a multi-stage approach to process and analyze news:

  • Real-Time News Crawlers: The system utilizes crawlers to collect news articles from 24 selected online media outlets, ensuring the database is kept up-to-date.
  • Event Clustering: Instead of analyzing articles individually, Islander groups them into "event clusters." This is achieved through a batch clustering process that identifies articles discussing similar topics within a short timeframe. A sliding window mechanism is used to maintain clusters over time, representing a specific event's coverage.
  • Event Title Generation: For each event cluster, an extractive summarization technique (TextRank) is applied to select the most representative title from the news articles.
  • Tag Generation: An abstractive tag generator, based on a GPT-like language model, is used to create descriptive tags for events. This model is trained on a large corpus of news articles with journalist-provided tags. The system also incorporates a strategy of combining predictions from multiple news sources to improve tag generation accuracy.
  • Metrics Measurement: Islander defines and measures three key metrics:
  • Suspicion Level: This metric is determined by factors like the incitation and bias of news titles, and the subjectivity of content. Incitation is assessed using a fine-tuned Chinese-BERT model on the CVAT dataset. Title bias is modeled as an aspect-level sentiment analysis task using RoBERTa. Content subjectivity is measured by identifying subjective words within news articles using a fine-tuned BERT model.
  • Popularity Level: This is measured by the size of the event cluster (amount of news coverage) and the volume of related public opinion.
  • Sentiment Level: A sentiment dataset is constructed from various sources (forums, social media, Twitter) and used to fine-tune a Chinese-based BERT model for sentiment scoring of both news content and user comments.
  • Suspicion Level of Public Opinions: The system also analyzes public opinions to detect coordinated behaviors that might indicate manipulation. It uses user embeddings learned via node2vec on a user network derived from co-occurrence relations and identifies bot-style writing by analyzing frequent phrases in comments.

Datasets

  • CVAT dataset: Used for predicting the incitation degree of sentences for title incitation assessment.
  • News articles with tags: A corpus of 0.7 million news pages with journalist-provided tags was used to train the tag generation model.
  • Taiwanese forum, Chinese social media, and Twitter posts/comments: These sources were used to construct the sentiment dataset for sentiment analysis.
  • Domain expert-labeled news content: 26 pieces of news content were labeled by domain experts to assess content subjectivity.

The paper does not explicitly mention the creation of new benchmark datasets, but rather the utilization and adaptation of existing ones for their specific tasks.

Key Insights and Ideas

  • Event-centric analysis is crucial: Moving beyond individual articles to cluster news around events provides a more holistic and efficient way to understand news coverage and public discourse.
  • Quality metrics beyond popularity: The paper highlights the importance of defining and measuring proxies for news quality (objectivity, neutrality, non-incitement) to counter the trend of clickbait and sensationalism.
  • Public opinion manipulation detection: Analyzing user co-occurrence and bot-style writing offers a novel approach to identifying coordinated behaviors that could skew public perception.
  • Leveraging pre-trained models for nuanced analysis: The effective fine-tuning of BERT and RoBERTa models for tasks like incitation detection, bias analysis, and sentiment scoring demonstrates the power of transfer learning in understanding complex linguistic nuances.
  • Combining multiple news sources improves tag generation: The finding that aggregating predictions from different news sources enhances tag generation accuracy suggests a robust strategy for dealing with diverse reporting styles.

Key Results

  • The tag generation model significantly outperforms TFIDF and TextRank baselines, especially for larger K values, indicating its ability to capture high-level concepts beyond simple keyword matching.
  • The proposed aspect-level sentiment classifier achieved an F1 score of 0.6922 with weak supervision, demonstrating its effectiveness in identifying journalist stance.
  • The system provides a web interface that allows users to browse trending topics, compare news metrics across different media, and analyze events over time, facilitating a more informed consumption of news.

Limitations and Future Work

  • Subjectivity of "quality": The paper acknowledges that the defined metrics (objectivity, neutrality, non-incitement) are proxies for quality and that a universally agreed-upon definition of "quality" is complex.
  • Data imbalance: The evaluation metric used (macro F1-score) addresses data imbalance, but the inherent scarcity of labeled data for some tasks (e.g., bias detection) remains a challenge.
  • Future improvements: The authors plan to continuously improve the system by incorporating new advances in natural language processing and exploring new data sources. They also suggest that the Islander modules can be further enhanced.

Summary updated

Abstract

With thousands of news articles from hundreds of sources distributed and shared every day, news consumption and information acquisition have been increasingly difficult for readers. Additionally, the content of news articles is becoming catchy or even inciting to attract readership, harming the accuracy of news reporting. We present Islander, an online news analyzing system. The system allows users to browse trending topics with articles from multiple sources and perspectives. We define several metrics as proxies for news quality, and develop algorithms for automatic estimation. The quality estimation results are delivered through a web interface to newsreaders for easy access to news and information. The website is publicly available at https://islander.cc/
BibTeX
@misc{huang2022islander,
  author = {Chao-Wei Huang and Kai-Chou Yang and Zi-Yuan Chen and Hao-Chien Cheng and Po-Yu Wu and Yu-Yang Huang and Chung-Kai Hsieh and Geng-Zhi Wildsky Fann and Ting-Yin Cheng and Ethan Tu and Yun-Nung Chen},
  title = {{Islander: A Real-Time News Monitoring and Analysis System}},
  year = {2022},
  eprint = {2204.11457},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.11457},
}
Figure