Assessing Arabic Weblog Credibility via Deep Co-learning

Chadi Helwe, Shady Elbassuoni, Ayman Al Zaatari, Wassim El-HajjACL 2019

TL;DR

This paper tackles the challenge of assessing the credibility of Arabic weblogs, a crucial task given the rise of misinformation. They introduce "deep co-learning," a semi-supervised deep learning approach that leverages a small labeled dataset and a larger unlabeled dataset to train robust credibility assessment models, outperforming traditional and fully-supervised deep learning methods.

What Problem Does This Paper Address?

The internet has become a primary source of news and information, with weblogs (blogs) playing an increasingly significant role. However, the proliferation of misinformation and disinformation poses a serious threat, impacting public opinion and even election outcomes. Accurately assessing the credibility of online content is therefore paramount. While research has explored credibility assessment for various online platforms like tweets, the specific domain of Arabic weblogs has been largely overlooked. A major hurdle in this area is the scarcity of labeled datasets suitable for training sophisticated machine learning models, especially deep learning models which typically require substantial amounts of data. This paper aims to bridge this gap by proposing a novel deep learning approach that can effectively learn from limited labeled data.

Methods

The core contribution of this paper is the deep co-learning approach, a novel adaptation of the co-training paradigm for deep learning in the context of credibility assessment. The method involves training two distinct deep neural network classifiers simultaneously. These classifiers are designed to learn from different "views" of the data.

  1. Initial Training: A small, fully-labeled dataset is used to bootstrap two convolutional neural network (CNN) models.
  2. Model 1: Utilizes pre-trained word-level embeddings (CBOW).
  3. Model 2: Employs character-level embeddings. Both CNNs share a similar architecture: multiple 1D convolutional layers, max pooling layers, a global max pooling layer, and a fully connected layer, culminating in a softmax output for classification.

  4. Semi-Supervised Iterative Retraining: The key innovation lies in how these models are iteratively refined using unlabeled data.

  5. A portion of the unlabeled data is randomly sampled.
  6. Each trained classifier independently predicts labels for these sampled instances.
  7. The most confidently predicted instances (top-k) from one classifier are then used to retrain the other classifier. This process of mutual learning, where each model leverages the other's predictions on unlabeled data, allows the models to improve their understanding of credibility without requiring explicit human labels for the bulk of the data.
  8. The ensemble average of the two models is evaluated on a validation set, and the models are updated if performance improves. This iterative process continues for a set number of iterations.

Datasets

  • Labeled Dataset: The authors utilize a dataset of Arabic blog posts constructed by Al Zaatari et al. (2016). This dataset contains 268 Arabic blog posts, labeled as "credible," "fairly credible," or "not credible." The posts cover trending topics such as elections, wars, and conflicts. The paper notes this is the only publicly available dataset for Arabic blog credibility assessment.
  • Unlabeled Dataset: A large corpus of 20,392 unlabeled Arabic blog posts was collected from Al Arabiya Blogs and Al Hudood. This dataset is crucial for the semi-supervised learning aspect of the deep co-learning approach.

Key Insights and Ideas

  • Leveraging Different Data Views: The success of deep co-learning hinges on the idea that different representations of the data (word-level vs. character-level embeddings) provide complementary information. By having two classifiers learn from these distinct views and teach each other, the model can achieve a more robust understanding of credibility.
  • The Power of Semi-Supervision for Scarce Data: The paper convincingly demonstrates that semi-supervised learning, specifically their deep co-learning adaptation, can significantly outperform fully-supervised methods when labeled data is scarce. This is a critical finding for many real-world NLP tasks where data annotation is expensive and time-consuming.
  • Character-Level Embeddings' Advantage: The study highlights the benefit of character-level embeddings, noting their ability to capture nuances like misspellings, emoticons, and n-grams, which can be indicative of credibility.
  • Ensemble Limitations with Insufficient Data: The paper shows that even a simple ensemble of word- and character-level CNNs performs worse than individual models when training data is limited, underscoring the need for more sophisticated techniques like co-learning.

Key Results

The deep co-learning approach achieved a F1-Score of 0.63 on the testing dataset. This significantly outperformed:

  • SVM TF-IDF: 0.57
  • Word-CNN (fully supervised): 0.52
  • Char-CNN (fully supervised): 0.54
  • Ensemble CNN (fully supervised): 0.50

The authors note that the SVM baseline, despite being a simpler model, performed better than the fully supervised deep learning models, likely due to the small size of the labeled training dataset. The deep co-learning approach, however, managed to overcome this limitation and achieve superior performance.

Limitations and Future Work

Limitations: * Small Labeled Dataset: The primary limitation acknowledged is the small size of the labeled dataset, which impacts the performance of fully-supervised models and necessitates the semi-supervised approach. * Computational Cost: The co-learning process, involving iterative retraining, is computationally intensive, with each iteration taking around 24 hours.

Future Work: * Extended Training: Train the deep co-learning approach for a longer duration to potentially further improve performance. * Publicly Available Labeled Data: Label more of the collected unlabeled blog posts using crowdsourcing and make this expanded dataset publicly available to benefit the research community. * Exploring New Architectures and Features: Experiment with different neural network architectures and incorporate additional linguistic features into the models.

Summary updated

BibTeX
@inproceedings{helwe2019assessing,
  author = {Chadi Helwe and Shady Elbassuoni and Ayman Al Zaatari and Wassim El-Hajj},
  title = {{Assessing Arabic Weblog Credibility via Deep Co-learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure