Profiling Reputation of Corporate Entities in Semantic Space

Jussi Karlgren, Magnus Sahlgren, Fredrik Olsson, Fredrik Espinoza, Ola HamforsCLEF (Online Working Notes/Labs/Workshop) 2012 · OpenReview

TL;DR

This paper presents Gavagai's system for the CLEF 2012 RepLab task, which aims to profile corporate reputation from streaming microblog data. Their approach leverages a "semantic space" model built on distributional semantics, achieving excellent performance on the task with standard settings.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically assessing corporate reputation from the vast and noisy stream of online data, specifically microblogs like Twitter. Traditional sentiment analysis often falls short because it might miss factual statements that impact reputation or misinterpret attitudinal statements that don't align with a company's desired image. The core problem is to accurately identify whether a microblog post refers to a specific company and, more importantly, whether it positively or negatively affects that company's reputation, going beyond simple positive/negative sentiment.

Methods

Gavagai's system, "Ethersource," is built upon a "semantic space" model derived from distributional semantics. This model represents words and their relationships in a high-dimensional vector space, allowing for nuanced understanding of meaning. For the RepLab task, they used a baseline system that processed streaming text. The system first filters microblog posts to identify those relevant to a target company using a combination of manual terms and a support system based on their semantic space model. Once a post is deemed relevant, it's then polarized with respect to predefined "customer satisfaction poles" (akin to positive/negative sentiment but tailored for reputation). The system uses editorially selected terms, augmented by their semantic space model, to define these poles. A threshold is applied to determine if a post has a significant effect on company reputation.

Datasets

The experimental data consisted of thirty-six sets of microblog posts filtered to contain a company name. The first six sets were used for training, and the remaining thirty were used for testing. The paper does not explicitly name the companies or provide details about the origin of these datasets beyond them being microblog posts.

Key Insights and Ideas

  • Beyond Simple Sentiment: The paper strongly emphasizes that reputation profiling is not just about positive or negative sentiment. Factual statements can impact reputation, and even strongly worded opinions might not affect it if they are about the "wrong" aspect of a company.
  • The Power of Semantic Space: Their reliance on a semantic space model, built from large-scale text analysis, allows for a more sophisticated understanding of word relationships and context, which is crucial for accurately profiling reputation.
  • Time Series Matters: The authors note that in commercial applications, they work with time series and sequences of posts rather than single posts. This is because the interpretation of a single post can be ambiguous and context-dependent, as illustrated by examples of potentially ambiguous tweets.

Key Results

Gavagai's baseline system achieved an overall profiling accuracy of 40.0%, which was calculated by correctly identifying both relevance and polarity. In the relevance assessment (filtering), their system achieved a filtering accuracy of 77.4%, demonstrating high reliability. For predicting the effect and direction of reputation impact, the system achieved an accuracy of 37%, with the threshold for considering a post impactful set conservatively.

Limitations and Future Work

The paper acknowledges that the analysis of individual posts can be subjective, as illustrated by examples where the interpretation of a tweet's impact on reputation depends heavily on external knowledge or perspective. The authors also mention that in their commercial applications, they prioritize analyzing posts in time series and sequences rather than in isolation, suggesting that this is a limitation of the task setup for real-world scenarios. Future work would likely involve further refining the semantic space model and exploring more sophisticated methods for handling the nuances of reputation impact in streaming data.

Summary updated

BibTeX
@inproceedings{karlgren2012profiling,
  author = {Jussi Karlgren and Magnus Sahlgren and Fredrik Olsson and Fredrik Espinoza and Ola Hamfors},
  title = {{Profiling Reputation of Corporate Entities in Semantic Space}},
  booktitle = {CLEF (Online Working Notes/Labs/Workshop)},
  year = {2012},
  url = {https://openreview.net/forum?id=U0SSnkLLKf},
}
Figure