Clarity: Data-Driven Automatic Assessment of Product Competitiveness

Sheema Usmani, Mariana Bernagozzi, Yufeng Huang, Michelle Morales, Amir Sabet Sarvestani, Biplav SrivastavaAAAI 2020 · OpenReview

Abstract

Competitive analysis is a critical part of any business, yet the process of scouring online content to understand market threats is traditionally labor-intensive and slow. This paper presents Clarity, a data-driven unsupervised system for assessing product competitiveness. Currently deployed at IBM, Clarity has been used by over 1,500 people to analyze more than 800 products. The system aggregates numeric ratings, sentiment across key performance dimensions, content volume, and recency to generate an explainable "Clarity Score." Results are visualized in an interactive dashboard, allowing users to track performance and understand the factors driving their market standing.

The Challenge of Manual Market Intelligence

In the modern business landscape, product managers and marketers must continuously monitor rivals to maintain a competitive edge. Traditionally, this involves manually searching forums, review sites, and news articles—a process that is expensive, prone to human bias, and often produces reports that are obsolete by the time they are finished. There is a clear need for a scalable, automated system that can monitor the competitive landscape in real-time and provide the "why" behind a product's market position.

The Clarity Pipeline

Clarity transforms unstructured text into quantitative metrics through a multi-stage pipeline:

  1. Data Aggregation: The system collects and standardizes reviews from various public forums.
  2. NLP Engine: Using the IBM Watson Natural Language Understanding (NLU) API, it extracts keywords and calculates sentiment.
  3. Driver Mapping: Keywords are mapped to five core drivers—Price, Compatibility, Functionality, Support, and Usability—based on the cosine distance between word vectors.
  4. Scoring & Weighting: Clarity uses percentile scoring to normalize data across the market and applies linear decay to prioritize recent reviews.
The NLP pipeline extracts keywords from reviews and maps them to product drivers using word vector distances.

Key Insights: Normalization and Explainability

A core innovation of Clarity is its use of percentile scoring. By ranking a product’s performance against the entire distribution of the marketplace, the system automatically handles the variance in scales across different data sources.

Furthermore, the system prioritizes explainability. A high-level competitiveness score is insufficient for business decisions; managers need to know which specific areas require attention. Clarity breaks down the aggregate score into the five performance drivers, enabling users to see if they are winning on "Functionality" but losing ground due to "Support" or "Price."

The Clarity dashboard provides a high-level score alongside granular breakdowns of key performance drivers like price and usability.

Interestingly, the authors noted a "human ceiling" in performance. Human experts only agree on topic classification about 55% of the time. Clarity’s model achieved a score of 0.51, suggesting it performs near the practical limit of natural language understanding for this specific task.

Validation and Results

The system's efficacy was validated across five distinct IT markets, including Cloud IaaS and IoT Platforms.

  • Expert Alignment: Clarity Scores showed strong alignment with Gartner Magic Quadrants. Products identified as "Leaders" by Gartner consistently maintained higher Clarity Scores than "Niche Players."
  • Customer Satisfaction: A positive correlation was found between Clarity Scores and Net Promoter Scores (NPS), confirming that the automated metric reflects real-world customer sentiment.
  • Operational Scale: The system has achieved significant internal adoption at IBM, supporting over 160 competitive analyses and generating 16,000+ unique views per quarter.

Limitations and Future Directions

While successful, Clarity faces challenges with linguistic nuances like sarcasm and the broad nature of its current drivers. Future work focuses on expanding the five drivers into 14 more specific sub-topics and transitioning toward supervised learning models by leveraging the feedback loop created by its large internal user base.

Summary updated

Abstract

Competitive analysis is a critical part of any business. Product managers, sellers, and marketers spend time and resources scouring through an immense amount of online and offline content, aiming to discover what their competitors are doing in the marketplace to understand what type of threat they pose to their business' financial well-being. Currently, this process is time and labor-intensive, slow and costly. This paper presents Clarity, a data-driven unsupervised system for assessment of products, which is currently in deployment in the large IT company, IBM. Clarity has been running for more than a year and is used by over 1,500 people to perform over 160 competitive analyses involving over 800 products. The system considers multiple factors from a collection of online content: numeric ratings by online users, sentiments of reviews for key product performance dimensions, content volume, and recency of content. The results and explanations of factors leading to the results are visualized in an interactive dashboard that allows users to track their product's performance as well as understand main contributing factors. Its efficacy has been tested in a series of cases across IBM's portfolio which spans software, hardware, and services.
BibTeX
@inproceedings{usmani2020clarity,
  author = {Sheema Usmani and Mariana Bernagozzi and Yufeng Huang and Michelle Morales and Amir Sabet Sarvestani and Biplav Srivastava},
  title = {{Clarity: Data-Driven Automatic Assessment of Product Competitiveness}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=nqidmk6R8F},
}
Figure