Opinion analysis for business intelligence applications

Adam Funk, Yaoyong Li, Horacio Saggion, Kalina Bontcheva, Christian LeiboldObi 2008 · DOI

TL;DR

This paper presents a machine learning system for opinion analysis that can classify text into positive/negative (binary) and 1-5 star ratings (multi-class). The system achieves good results using simple linguistic features and a Support Vector Machine (SVM), demonstrating its potential for business intelligence applications by extracting valuable sentiment data from unstructured online sources.

What Problem Does This Paper Address?

In today's digital age, businesses are inundated with unstructured text data from online forums, reviews, and social media. This data contains valuable opinions and sentiments about companies, products, and services. However, manually analyzing this vast amount of information is impractical. Business intelligence (BI) applications, such as credit rating or reputation management, require efficient ways to extract and understand these opinions. This paper addresses the challenge of automatically analyzing this text to identify and quantify opinions, making them usable for BI.

Methods

The authors developed a supervised machine-learning system using Support Vector Machines (SVMs). They focused on extracting simple linguistic features from text, such as the token's raw string, its lemmatized root form, and its part-of-speech (POS) tag. Crucially, they avoided relying on pre-defined lexicons of "opinion words," instead allowing the SVM to learn the sentiment-bearing features directly from the annotated data.

The system was applied to two main tasks:

  1. Binary Classification: Classifying text as either "thumbs-up" (positive) or "thumbs-down" (negative).
  2. Five-Way Classification: Classifying text into five discrete rating categories (1-star to 5-star).

They employed 10-fold and 5-fold cross-validation for training and evaluation, respectively.

Datasets

The paper describes the use of two distinct datasets:

  1. Consumer Forum Dataset (Binary Classification): A corpus of 92 HTML documents collected from a consumer forum. Each document contained a user's comment (paragraph of text) and an associated "thumbs-up" or "thumbs-down" rating, indicated by an image tag. This dataset was imbalanced, with 67% thumbs-down ratings.
  2. Consumer Forum Dataset (Five-Way Classification): A larger corpus of 600 HTML pages from another consumer forum. Each page contained one or more comments and a user's rating from 1-star to 5-star. This dataset contained approximately 7300 classification instances and was also imbalanced, with a strong bias towards 5-star ratings (67.9%).

Key Insights and Ideas

  • Simplicity is Powerful: The study demonstrates that even basic linguistic features (like word roots and POS tags) combined with SVMs can yield strong results in opinion classification, without needing complex linguistic resources or pre-defined sentiment lexicons.
  • Data Imbalance is a Challenge: The inherent bias in the datasets (more negative reviews in binary, more high ratings in multi-class) significantly impacted classification performance, particularly for the less frequent classes. This highlights the need for robust handling of imbalanced data in real-world applications.
  • Mid-Range Ratings are Fuzzy: Classifying intermediate ratings (2-star, 3-star) proved more difficult than classifying extreme ratings (1-star, 5-star). This is attributed to the subjective and less clearly defined nature of opinions associated with these mid-range scores.
  • Feature Engineering Trade-offs: Increasing the number of features or the size of n-grams (e.g., bigrams, trigrams) did not consistently improve performance and could even lead to sparsity issues and decreased accuracy, a common observation with SVMs.

Key Results

  • Binary Classification: The system achieved an overall accuracy of 80% using the "root, orth" (lemmatized root and orthographic features) for binary classification. This is considered a good benchmark, comparable to previous work.
  • Five-Way Classification: The system achieved a reasonable overall accuracy of 74% for the 1-5 star classification. However, performance was uneven, with high accuracy for 1-star (79.9%) and 5-star (85.2%) ratings, but significantly lower accuracy for 2-star (1.8%) and 3-star (5.8%) ratings.
  • The authors noted that their system tended to err in the negative direction for binary classification, misclassifying positive texts as negative more often, which they attributed to the dataset's bias.

Limitations and Future Work

Limitations:

  • Dataset Bias: The uneven distribution of ratings in both datasets significantly affected the performance, particularly for the less represented classes.
  • Mid-Range Ambiguity: The inherent fuzziness and less distinct nature of opinions in mid-range ratings (2-4 stars) made them harder to classify accurately.
  • Focus on Text: The current work primarily focuses on text-based sentiment and does not explicitly incorporate other modalities or contextual information.

Future Work:

  • Opinion Holder and Subject Identification: Extending the system to identify who is expressing the opinion and what the opinion is about (e.g., specific product features) using Named Entity Recognition (NER) and coreference resolution.
  • Ontology Integration: Developing a system to classify texts according to a "qualitative variable" category from a specially developed ontology for Business Intelligence (BI). This will enable the generation of RDF statements to populate a BI knowledge base.
  • Active Learning and Weak Supervision: Investigating techniques like SVM active learning to minimize training data requirements and exploring weakly supervised methods (e.g., probabilistic models for hedge classification) and transductive SVM for unsupervised learning.
  • Broader Application: Applying the developed techniques to a wider variety of text sources relevant to business intelligence and integrating them into the MUSING project's BI system.

Summary updated

BibTeX
@inproceedings{funk2008opinion,
  author = {Adam Funk and Yaoyong Li and Horacio Saggion and Kalina Bontcheva and Christian Leibold},
  title = {{Opinion analysis for business intelligence applications}},
  booktitle = {Obi},
  year = {2008},
  doi = {10.1145/1452567.1452570},
}
Figure