A Factored Neural Network Model for Characterizing Online Discussions in Vector Space

Hao Cheng, Hao Fang, Mari OstendorfEMNLP 2017 · OpenReview

Introduction

In online discussion forums like Reddit, the impact of a comment is determined not only by its literal content but also by its stylistic alignment with the community's culture and its ability to provoke responses. Traditional text representation methods, such as Doc2Vec or Bag-of-Words, often fail to capture these nuances, treating text as an isolated "black box." This research introduces a factored neural network model designed to learn unsupervised embeddings that explicitly separate community-wide stylistic "modes" from specific comment content, while also accounting for the social dynamics of multi-party discussions.

Methodology: The Factored Neural Network

The core innovation of this research is the factorization of comment embeddings into two distinct components: a Local Mode Vector and a Content Vector. This separation allows the model to distinguish between the general "vibe" of a subreddit and the unique information being communicated.

Architecture of the Factored Neural Network, showing the separation of global modes, content vectors, and response triggers.

As illustrated in Figure 1, the model employs an attention mechanism to select from a set of $K$ latent "global modes"—learned representations of community styles or topics. A residual learning architecture then determines the balance between these modes and the specific content needed to predict the next word in a sequence. Furthermore, the model incorporates a Response Generation component. By training the model to predict the words in a subsequent reply using an encoder-decoder framework, the resulting embeddings are forced to capture "response trigger" features—linguistic cues that incite engagement.

Experimental Setup

The model was evaluated using a massive dataset of Reddit threads from three subreddits with distinct cultures:

  • r/AskWomen (814,000 comments)
  • r/AskMen (1,057,000 comments)
  • r/Politics (2,180,000 comments)

The primary evaluation task was predicting a comment's "karma" score (up-votes minus down-votes), which serves as a proxy for community endorsement. Karma scores were quantized into eight discrete levels for this classification task.

Key Results and Performance

The factored model consistently outperformed standard baselines, including Bag-of-Words (BoW), Latent Dirichlet Allocation (LDA), and Doc2Vec. In the r/Politics dataset, the model achieved an average F1 score of 54.8%, significantly higher than the 53.0% achieved by Doc2Vec and the 51.3% baseline using only metadata.

Performance comparison across r/AskWomen, r/AskMen, and r/Politics, showing the effectiveness of the factored approach.

Ablation studies confirmed that both the "Mode" and "Response" components are critical to the model's success; removing either component led to a measurable decrease in prediction accuracy.

Interpretability and Insights

Beyond quantitative gains, the factorization provides qualitative insights into community dynamics:

  • Latent Modes: The learned modes correspond to specific behaviors. For example, in r/Politics, certain modes captured the use of rhetorical questions for disagreement or aggressive attention-grabbing language.
  • Structural Positioning: The model revealed that stylistic elements (like expletives or rhetorical questions) are often "front-loaded" at the start of comments, whereas topical content is more evenly distributed.
  • Social Dynamics: By modeling response triggers, the system learned to distinguish between comments likely to receive complimentary versus disagreeing replies.

Conclusion

This factored neural network approach demonstrates that the structural context of online forums—specifically the relationship between a comment and its subsequent replies—provides a powerful, unsupervised signal for learning rich text representations. By separating style from content, the model achieves both superior predictive performance and enhanced interpretability. Future work may extend these embeddings to more complex tasks such as dialogue act classification or argument persuasiveness ranking.

Summary updated

BibTeX
@inproceedings{cheng2017factored,
  author = {Hao Cheng and Hao Fang and Mari Ostendorf},
  title = {{A Factored Neural Network Model for Characterizing Online Discussions in Vector Space}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=HyZ-OMMdbr},
}
Figure