Signals of Expertise in Public and Enterprise Social Q&A

Shimei Pan, Elijah Mayfield, Jie Lü, Jennifer C. Lai

Abstract

The success of internet social question-answering (Q&A) sites has led businesses to attempt to duplicate this model internally. In such systems, demand for expertise is high, meaning that finding high-quality answers quickly is a top priority. Prior work on expertise discovery, though, has focused on high-traffic public websites like StackExchange. Here, we show that the most predictive metadata features from public websites do not transfer well to the enterprise.

Introduction

Social Q&A platforms like StackOverflow and Quora have revolutionized knowledge sharing on the public web. Naturally, enterprises seek to replicate this success internally to streamline information flow. However, identifying "experts" and high-quality content in a corporate setting presents unique challenges. This research addresses the domain transfer problem: can the signals that identify a "good" answer on the open web be effectively applied within a private company?

Methodology: Feature Clusters

To evaluate the effectiveness of various signals, the researchers categorized metadata into five distinct "feature clusters" based on prior literature:

  1. Asker Cluster: Metadata regarding the question's author.
  2. Answerer Cluster: Historical activity and behavior of the respondent.
  3. Context Cluster: Environmental data such as view counts, comments, and timing.
  4. Shallow Text Cluster: Basic structural metrics like word count and HTML formatting.
  5. Shallow Style Cluster: Linguistic markers, including pronoun usage and "trust" keywords.
The five feature clusters used to categorize metadata signals for predicting answer quality.

The study compared two datasets: Webmasters StackExchange (Public) and IBM Answers (Enterprise). While thematically similar, the public site exhibited significantly higher traffic and engagement levels, with nearly four times the average votes per answer compared to the enterprise system.

Key Results: The Failure of Domain Transfer

The core finding of the study is that most features used to predict quality on public sites perform significantly worse in an enterprise context. The researchers measured the correlation between these clusters and two outcomes: Answer Acceptance (asker satisfaction) and Community Votes (reader satisfaction).

Comparison of predictive power (correlation) across clusters for Public and Enterprise datasets, showing significant drops in most categories.

As shown in the results, the Context and Shallow Text clusters saw their predictive power for community votes plummet by 61%. This indicates that the high-traffic "rising tide" effect of public forums—where popular questions attract votes regardless of specific answer quality—is much less prevalent in the enterprise. Conversely, the Answerer Cluster was the only category that performed better in the enterprise, showing a 130% increase in predictive power for community votes.

Insights: Bursty Expertise and Traffic Noise

The research highlights two critical differences between the domains:

  • The "Burst" Phenomenon: In corporate environments, long-term reputation is a poor indicator of quality. Instead, the strongest signal is a user's activity within the last 7 days. Enterprise experts often contribute in "bursts" triggered by specific needs or search results, rather than maintaining a consistent habit of participation.
  • Presentation vs. Content: On StackExchange, visual formatting (like code blocks) strongly predicts community votes. In the enterprise, this effect is negligible, suggesting that corporate users prioritize raw information over presentation style.

Conclusion

The study concludes that enterprise Q&A systems cannot simply copy the reputation and ranking algorithms of public sites. To effectively identify expertise in a corporate setting, systems must be "burst-aware," focusing on recent activity rather than static metadata or long-term history. Relying on traditional public-web signals will likely result in missing the most relevant internal experts.

Summary updated

Abstract

The success of internet social question-answering (Q&A) sites has led businesses to attempt to duplicate this model in- ternally. In such systems, demand for expertise is high, mean- ing that finding high-quality answers quickly is a top priority. Prior work on expertise discovery, though, has focused on high-traffic public websites like StackExchange. Here, we show that the most predictive metadata features from public websites do not transfer well to the enterprise.
BibTeX
@inproceedings{pan2021signals,
  author = {Shimei Pan and Elijah Mayfield and Jie Lü and Jennifer C. Lai},
  title = {{Signals of Expertise in Public and Enterprise Social Q\&A}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/icwsm.v9i1.14646},
}
Figure