Automatic Detection of Point of View Differences in Wikipedia

Khalid Al Khatib, Hinrich Schütze, Cathleen KantnerCOLING 2012 · OpenReview

Abstract: Khalid Al Khatib, Hinrich Schütze, Cathleen Kantner. Proceedings of COLING 2012.

2012.

Introduction

In the world of journalism and encyclopedias, "objectivity" is the gold standard. However, even when writers adhere to a "Neutral Point of View" (NPOV), bias often creeps in through two subtle channels: framing and selection. Framing involves choosing different ways to describe the same fact (e.g., "his wars caused deaths" vs. "people died in his wars"), while selection refers to choosing which facts to include or omit due to cultural relevance.

While traditional sentiment analysis focuses on subjective text like movie reviews, this research tackles the harder problem of detecting bias in ostensibly objective text. The authors propose an automated method to detect and quantify these "Point of View" (POV) differences between Wikipedia articles on the same topic in different languages.

Methodology

The authors propose a two-step pipeline to move from individual sentences to a relative comparison of documents:

  1. Absolute POV Classification: A Maximum Entropy (MaxEnt) classifier was trained to label sentences as Positive, Negative, or Neutral relative to a specific target. Features included a combination of Bag-of-Words (BOW) and letter $k$-grams. For Arabic, which is highly inflected, the researchers utilized light stemming and root-based lemmatization to improve accuracy.
  2. Document-Level Aggregation: They defined a POVScore for a document as the average difference between the probability of a sentence being positive vs. negative.
  3. Relative POV Difference ($\Delta_{POV}$): Instead of defining a "perfectly neutral" baseline, they calculated the difference between the POVScores of two documents (e.g., the English vs. Arabic Wikipedia articles for the same person). This shift from absolute to relative assessment allows the system to focus on the delta between cultural perspectives.

Datasets and Annotation

The study utilized two primary datasets. The first was a sentence-level gold standard consisting of 1,200 sentences from Arabic Wikipedia and 1,200 from English Wikipedia, annotated via Amazon Mechanical Turk (AMT). The second was a document-level gold standard of 30 pairs of Wikipedia articles covering Israeli nationalists, Arab nationalists, and international celebrities, annotated by bilingual experts.

Inter-annotator agreement and label distribution for the sentence-level gold standard.

As shown in the data above, inter-annotator agreement was generally higher for English than for Arabic, likely due to the linguistic complexities of the latter. The distribution of labels indicates that while neutral and positive sentiments dominate, there is a significant enough presence of negative framing to allow for comparative analysis.

Key Results and Insights

  • Classifier Performance: The MaxEnt classifier significantly outperformed the baseline. For Arabic, the best results came from using stems with letter n-grams (Accuracy: 0.584), while for English, n-grams on tokens performed best (Accuracy: 0.608).
  • POV Trends: The system successfully identified cultural biases. For example, Israeli personalities generally had more negative POVScores in Arabic Wikipedia than in English, while Islamic/Arabic personalities were framed more positively in Arabic.
  • Cultural Enthusiasm: Interestingly, the data revealed that Arabic Wikipedia tends to be more "enthusiastic" about international artists and athletes compared to English Wikipedia, resulting in higher positive POVScores for neutral international figures.

Limitations and Future Work

The authors acknowledge the limitations of their current Bag-of-Words (BOW) approach. A BOW model cannot handle negation or complex phrasing; for example, "John started a war on violence" contains negative words but reflects a positive POV toward John. Additionally, the current system assumes every sentence in an article is about the main subject, which can lead to incorrect sentiment attribution. Future work will involve more sophisticated representations that account for word scope and use language understanding to ensure sentiment is correctly linked to the intended target.

Summary updated

BibTeX
@inproceedings{khatib2012automatic,
  author = {Khalid Al Khatib and Hinrich Schütze and Cathleen Kantner},
  title = {{Automatic Detection of Point of View Differences in Wikipedia}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
  url = {https://openreview.net/forum?id=2N1-7nowZ0X},
}
Figure