Message Impartiality in Social Media Discussions

Muhammad Bilal Zafar, Krishna P. Gummadi, Cristian Danescu-Niculescu-MizilAAAI 2021 · DOI

Abstract

Discourse on social media platforms is often plagued by acute polarization, with different camps promoting different perspectives on the issue at hand. While previous research has focused on the affiliation of groups and individuals, we propose a new finer-grained perspective: studying the impartiality of individual messages. We operationalize message impartiality in terms of how discernible the affiliation of its author is and introduce a methodology for quantifying it automatically. Our framework enables us to study the effects of (im)partiality on social media discussions at scale. We show that partial messages are twice as likely to spread than impartial ones, even after controlling for author and topic.

Introduction

Most research on social media polarization focuses on the author—labeling users as "liberal" or "conservative." However, this ignores the nuance of individual utterances. A staunch partisan can still post a neutral news update, and a moderate can post a highly biased "hot take." The challenge is that "impartiality" is deeply subjective and hard for humans to annotate consistently. This paper seeks to operationalize impartiality as a measurable metric and create an automated system that works for emerging events without needing new human labels every time.

Methodology: Affiliation Discernibility

The core of our approach is the concept of Affiliation Discernibility. If a message is truly impartial, an observer should not be able to tell which "side" the author supports. Conversely, if a message contains cues that make the author's affiliation obvious, it is considered partial.

We implement this using the Unigram Method. This technique involves scoring individual words (unigrams) based on how strongly they signal a specific affiliation within a given context. For instance, during the 2013 U.S. government shutdown, terms like "#GOPShutdown" were strong signals for Democrats. A message’s partiality score ($mp\text{-}score$) is determined by its most "discerning" word; if a tweet contains even one highly partisan keyword, the entire message is flagged as partial.

Validation of the Impartiality Metric

To ensure our automated metric reflects human perception, we compared the automated scores against human judgments obtained via Amazon Mechanical Turk. The results showed a strong alignment between the "discernibility" metric and human intuition regarding bias.

Correlation between human agreement on bias (Agreement bin) and the automated partiality rating for the 2013 Government Shutdown.

As shown in Figure 1, as the level of human agreement on a message's bias increases (the agreement bin), the average partiality rating generated by our model also increases, validating the metric's accuracy.

Performance and Results

Our analysis across seven major polarized events—including political debates and sports championships—revealed several key findings.

Methodological Superiority

The proposed Unigram method significantly outperformed standard supervised learning models, such as Naive Bayes and Bagged Trees, particularly when applied to "emerging events" where prior labeled data is scarce.

Comparison of the proposed Unigram method against standard machine learning baselines in predicting partiality ratings.

In Figure 4, we observe that the Unigram method maintains a much higher correlation with human partiality ratings across different score bins compared to baseline models.

The Virality of Partiality

The most striking result is the "virality gap." Partial messages are twice as likely to be retweeted as impartial ones. This holds true even after controlling for the author's follower count and the specific topic being discussed, suggesting that bias itself is a primary driver of engagement on social media.

Temporal Dynamics

Impartiality is not a static property of a discussion. During the 2013 government shutdown, discourse began relatively neutral but became sharply more partial during key legislative moments, such as the proposal of major amendments.

Conclusion and Future Work

This work provides a new lens for studying online polarization by focusing on the impartiality of individual messages rather than just author identity. While our current model is robust, it faces limitations with linguistic nuances like sarcasm and irony, which can confuse unigram-based detection. Future research will explore more complex linguistic features, such as discourse markers and hedges, to better capture the subtleties of impartial speech in multi-factional real-world conflicts.

Summary updated

Abstract

Discourse on social media platforms is often plagued by acute polarization, with different camps promoting different perspectives on the issue at hand—compare, for example, the differences in the liberal and conservative discourse on the U.S. immigration debate. A large body of research has studied this phenomenon by focusing on the affiliation of groups and individuals. We propose a new finer-grained perspective: studying the impartiality of individual messages. While the notion of message impartiality is quite intuitive, the lack of an objective definition and of a way to measure it directly has largely obstructed scientific examination. In this work we operationalize message impartiality in terms of how discernible the affiliation of its author is, and introduce a methodology for quantifying it automatically. Unlike a supervised machine learning approach, our method can be used in the context of emerging events where impartiality labels are not immediately available. Our framework enables us to study the effects of (im)partiality on social media discussions at scale. We show that this phenomenon is highly consequential, with partial messages being twice more likely to spread than impartial ones, even after controlling for author and topic. By taking this fine-grained approach to polarization, we also provide new insights into the temporal evolution of online discussions centered around major political and sporting events.
BibTeX
@inproceedings{zafar2021message,
  author = {Muhammad Bilal Zafar and Krishna P. Gummadi and Cristian Danescu-Niculescu-Mizil},
  title = {{Message Impartiality in Social Media Discussions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/icwsm.v10i1.14754},
}
Figure