Abstract
In computational argumentation, generating counter-arguments typically relies on explicit negation. However, human discourse often employs "semantic edits"—replacing specific phrases to shift a claim's stance or suggest alternatives. We present a system developed at Columbia University that leverages a novel dataset of over 1 million Reddit "Fixed That For You" (FTFY) comments to train neural models for context-aware contrastive claim generation. Our results demonstrate that models using specialized decoding constraints and subreddit-level context produce more coherent and contrastive responses than standard sequence-to-sequence baselines.
Introduction: Beyond Simple Negation
Generating a counter-argument usually begins with a claim—a statement in dispute. Most automated systems approach this through explicit negation (e.g., changing "Healthcare is a right" to "Healthcare is not a right"). While logically sound, this approach fails to capture the nuance of human debate, where productive dialogue often involves offering alternative suggestions or shifting viewpoints entirely. For example, a more natural contrast to "Healthcare is a right" might be "Healthcare is a commodity" rather than a simple negation.
The challenge lies in the complexity of these "contrastive claims." They require a deep understanding of which words to change and what semantically relevant replacements to use. Furthermore, high-quality, large-scale data for this specific task has historically been scarce.
The FTFY Corpus and Dataset Construction
To address the data scarcity, the researchers introduced the FTFY Corpus, mined from Reddit. They scraped approximately 1.08 million pairs of comments where a user responded to another using the acronym FTFY (Fixed That For You). This serves as a form of "distant labeling," where users have self-identified their edits as corrections or alternative viewpoints.
Because the FTFY tag is also used for jokes and typo corrections, a filtering step was necessary. The authors trained a logistic regression classifier to identify true "contrastive claims," achieving an 80.25 F-score. For high-quality evaluation, they also curated an AMT Gold Standard set of 2,625 comments, each paired with five human-generated contrastive responses.
Methodology: Semantic Editing via Seq2Seq
The problem is framed as a sequence-to-sequence (Seq2Seq) task, similar to machine translation, but with several architectural enhancements:
- Encoder-Decoder with Attention: A bi-directional GRU encodes the original claim, and a GRU decoder generates the "fixed" version.
- The "Counter" Mechanism: To prevent the model from simply copying input or generating incoherent lengths, a counter variable forces the generation of a specific number of "new" content words before the sentence can conclude.
- Subreddit Embeddings: To handle domain-specific language (e.g., "bug" in a tech forum vs. a nature forum), the model incorporates subreddit-level context to disambiguate meanings.
- Multi-task Learning: A secondary loss term trains the encoder to predict whether a specific token in the input should be copied, deleted, or replaced.
Key Results and Evaluation
The "Best" model—incorporating the counter mechanism, subreddit embeddings, and copy loss—was compared against standard Seq2Seq baselines using both automated and human metrics.
On the Reddit test set, the best model achieved a BLEU-4 score of 23.13, significantly outperforming the baseline (18.81). It also showed higher "novelty" scores, indicating a successful shift from simple copying to active editing. Human evaluation on a 1-5 Likert scale further validated the approach, with the model outperforming the baseline across all metrics.
The results indicate that while the model is slightly more fluent, its primary strength is generating meaningful contrasts that remain coherent with the original statement.
Limitations and Future Directions
Despite its success, the model faces several challenges. It occasionally struggles with low-frequency named entities and can exhibit repetition due to the constrained decoding process. Furthermore, the model lacks "intention" or a specific stance; it might generate a conservative or liberal edit randomly. Future work may involve "controllable generation" to specify a desired stance and the integration of deeper world knowledge to handle complex contextual contrasts.
Summary updated