Overview of RepLab 2012: Evaluating Online Reputation Management Systems

Enrique Amigó, Adolfo Corujo, Julio Gonzalo, Edgar Meij, Maarten de Rijke

TL;DR

RepLab 2012 was the first competitive evaluation campaign for Online Reputation Management (ORM) systems, focusing on company reputation via Twitter data. It introduced novel tasks and a gold standard dataset annotated by ORM experts, aiming to advance automated ORM by evaluating systems on their ability to monitor and profile company reputations.

What Problem Does This Paper Address?

In today's hyper-connected world, a company's online reputation is paramount. Information, including public opinions, spreads rapidly, making it crucial for organizations to effectively manage their image. Online Reputation Management (ORM) encompasses activities to build, protect, and repair this image. While opinion mining has advanced, applying it to companies is more complex than to products because company reputations aren't tied to fixed features. This paper addresses the need for systematic evaluation of automated systems designed to assist in ORM, particularly in processing the vast and dynamic stream of information from social media like Twitter. The challenge lies in developing systems that can accurately identify relevant information, understand its sentiment and implications for reputation, and prioritize it for human experts.

Methods

RepLab 2012 defined two core tasks:

  • Profiling: This task involved annotating tweets for relevance (is the tweet about the company?) and polarity for reputation (does the tweet have positive, negative, or neutral implications for the company's image?). This differs from standard sentiment analysis by considering factual statements and the perspective of different stakeholders.
  • Monitoring: This task required systems to cluster recent tweets thematically and then order these clusters by priority, indicating which topics demand immediate attention for reputation management.

The evaluation employed novel metrics like Reliability and Sensitivity, combined into a harmonic mean (F(R,S)), and Accuracy. For monitoring, a combined measure of clustering and priority was developed.

Datasets

The RepLab 2012 campaign utilized Twitter data in both English and Spanish.

  • Trial Dataset: Contained at least 30,000 tweets per company for six companies.
  • Test Dataset: Was significantly larger, comprising tweets for 31 different companies, with 19,400 to 50,000 tweets per company.

A key aspect was the creation of a "gold standard" dataset, manually annotated by reputation management experts from Llorente & Cuenca. This dataset was crucial for evaluating system performance and gaining insights into the nuances of reputation management from a professional perspective. The dataset was split into "labelled" (annotated) and "background" (unannotated) sets.

Key Insights and Ideas

  • Polarity for Reputation is Distinct from Sentiment Analysis: The paper emphasizes that a tweet's implication for a company's reputation can differ from its sentiment. For example, a sad tweet about a celebrity's death might have negative sentiment but a positive implication for their reputation.
  • ORM is a Challenging Domain for Automation: The results indicate that current automated systems still have a significant gap to bridge in accurately performing ORM tasks, especially with limited training data.
  • Expert Annotations are Crucial but Costly: The use of professional ORM experts for annotation provided high-quality data but highlighted the expense and potential for inter-annotator disagreement, suggesting a need for more consistent annotation guidelines.
  • Twitter's Dynamic Nature Poses Challenges: The ephemeral nature of tweets and the inability to redistribute them directly complicates the long-term usability and comparability of test collections based on social media.

Key Results

  • Profiling Task: The top-performing systems achieved accuracies around 41% for the combined filtering and polarity task, indicating that automated profiling is far from solved. The F(R,S) scores for polarity were generally higher than for filtering, suggesting that determining reputation polarity might be slightly less challenging than identifying relevant tweets.
  • Monitoring Task: Systems performed below the baseline in clustering and priority assignment, with the best systems achieving F(R,S) scores around 0.29, while the baseline reached 0.41. This suggests that current clustering and prioritization techniques are not yet robust enough for this complex task.
  • Language Differences: Spanish presented a greater challenge than English, likely due to a comparative lack of language analysis tools for sentiment analysis and Twitter-specific language nuances.

Limitations and Future Work

  • Small Training Sets: The limited size of the training datasets was a deliberate choice to prevent over-reliance on machine learning and encourage more generalized solutions, but it also made the tasks more challenging.
  • Dynamic Nature of Twitter Data: The inability to redistribute tweets directly and the potential for tweets to be removed or changed over time makes test collections less stable and comparisons over time difficult.
  • Inter-Annotator Agreement: Differences in how experts annotated certain statements suggest a need for more consistent guidelines for reputation management reporting.

Future work could involve:

  • Focusing on Scenarios with More Training Data: Exploring settings where ample annotated material is available, mimicking typical agency-client relationships.
  • Continuous Learning Systems: Developing systems that can learn and adapt from annotated streams of data in real-time.
  • Improving Annotation Consistency: Further research into creating more standardized annotation guidelines for ORM.

Summary updated

BibTeX
@inproceedings{amigo2012overview,
  author = {Enrique Amigó and Adolfo Corujo and Julio Gonzalo and Edgar Meij and Maarten de Rijke},
  title = {{Overview of RepLab 2012: Evaluating Online Reputation Management Systems}},
  booktitle = {CLEF (Online Working Notes/Labs/Workshop)},
  year = {2012},
}
Figure