Same Author or Just Same Topic? Towards Content-Independent Style Representations

Anna Wegmann, Marijn Schraagen, Dong NguyenACL 2022

TL;DR

This paper tackles the challenge of learning "style" representations for text that are independent of the text's content. They propose a novel "Contrastive Authorship Verification" (CAV) setup and a "content control" (CC) mechanism using conversation context to train models that better disentangle style from topic, outperforming previous methods.

What Problem Does This Paper Address?

In natural language processing, understanding and generating text with specific stylistic nuances is crucial for various applications, from author profiling to machine translation. A common approach to training models for style representation is to leverage "authorship verification" (AV) tasks, which aim to determine if two texts were written by the same author. The underlying assumption is that the same author will exhibit a similar writing style. However, this assumption can be problematic because authors often write about similar topics, leading to style representations that inadvertently capture content information rather than just style. This paper addresses this "style vs. content" entanglement, aiming to create style representations that are truly independent of what the text is about.

Methods

The authors introduce two main methodological contributions:

  • Contrastive Authorship Verification (CAV) Setup: This is a variation of the traditional AV task. Instead of just comparing two texts, CAV involves an anchor text ($A_1$) and two other texts ($A_2$ and $B$). $A_2$ is from the same author as $A_1$, while $B$ is from a different author. The model must identify which of $A_2$ or $B$ is from the same author as $A_1$. This setup allows for learning objectives that use three input sentences, which have proven effective in semantic embedding learning.
  • Content Control (CC) via Topic Proxies: To disentangle style from content, the authors experiment with different levels of content control. They use "topic proxies" to approximate content. The primary proxy is conversation context: they assume texts from the same conversation are likely to be about the same topic. They compare this with using domain labels (e.g., subreddits) and a no content control baseline. The idea is that by forcing the model to distinguish between texts from different authors within the same conversation, it must rely more on stylistic cues than on topic similarity.

They train siamese BERT-based neural networks using contrastive and triplet loss functions on these modified AV tasks.

Datasets

The primary dataset used is a sample of Reddit conversations. Specifically, they use data from 100 active subreddits, sampled to include conversations with at least 10 posts. This dataset is suitable because conversations inherently provide a natural "topic proxy." They also leverage the STEL framework (Style Evaluation Task) for evaluating the learned style representations, particularly for their content-independence.

Key Insights and Ideas

  • Conversation context is a powerful content proxy: Using same-conversation utterances as a proxy for content control proved more effective than domain labels or no control. This suggests that fine-grained topical similarity within a conversation forces models to focus on stylistic differences.
  • AV alone isn't enough for good style representations: High performance on the standard AV task doesn't guarantee that the learned representations are truly style-focused. Models can achieve good AV scores by latching onto content correlations.
  • Disentangling style and content is challenging but achievable: The proposed CAV setup with conversation CC shows promise in learning representations that are more independent of content.
  • Style representations can capture subtle features: Agglomerative clustering revealed that the learned representations can distinguish subtle stylistic features like punctuation usage (e.g., single vs. double quotation marks) and casing.

Key Results

  • Improved Content-Independence: Models trained with the CAV setup and conversation CC achieved the best performance on a novel "STEL-Or-Content" task designed to test content-independence. This task showed a significant improvement from a random baseline (accuracy around 0.05) to up to 0.42 ± 0.01 for the best models.
  • AV Performance: While the CAV and AV setups led to similar performance on the AV task itself, the models trained with conversation CC generally performed best on the conversation test set, indicating better generalization within that context.
  • STEL Performance: Performance on the STEL task (measuring style dimensions) was comparable across models, but the authors noted a drop compared to the original RoBERTa base model, suggesting that fine-tuning for AV might sometimes "unlearn" certain style information.
  • Clustering: Agglomerative clustering showed that 46.2% of utterance pairs from the same author were grouped into the same cluster, significantly outperforming random assignment (20.1%). This indicates that the learned representations capture meaningful stylistic similarities.

Limitations and Future Work

  • Transferability of Conversation CC: The authors acknowledge that conversation labels are readily available in corpora like Reddit but transferring this "conversation CC" to datasets without inherent conversational structure remains a challenge.
  • Semantic Embeddings for Content Control: They propose using semantic sentence embeddings as a new content control level, where utterances are considered to have the same content if their embeddings are close.
  • STEL-Or-Content Task Difficulty: The "triplet problem" in the STEL-Or-Content task, where stylistic similarity might be ambiguous, limits performance. Future work could involve removing problematic instances.
  • Individual vs. Group Style: The models learn from thousands of authors, potentially capturing consistencies across groups of authors (e.g., demographic groups) rather than just individual idiolects. Exploring how different CC levels influence the types of styles learned is a future direction.
  • Demographic Skew: The use of Reddit data means the learned representations might be skewed towards younger, male demographics, potentially not being representative of all English language users.

Summary updated

BibTeX
@inproceedings{wegmann2022same,
  author = {Anna Wegmann and Marijn Schraagen and Dong Nguyen},
  title = {{Same Author or Just Same Topic? Towards Content-Independent Style Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure