ArgU: A Controllable Factual Argument Generator

Sougata Saha, Rohini K. SrihariACL 2023 · OpenReview

Abstract

While Large Language Models (LLMs) excel at generating fluent text, they often struggle with logical consistency and rhetorical control in automated argumentation. We introduce ArgU, a framework designed to move neural argument generation toward logical and factual "controllability." By leveraging classical argumentation theory and a novel two-step generation process, ArgU produces arguments that follow specific reasoning patterns. Our approach is supported by a new corpus of nearly 70,000 annotated arguments, the largest of its kind for identifying complex reasoning patterns.

Introduction

The core challenge in automated argumentation is controllability. Most existing generators allow users to specify a topic or sentiment, but they lack the precision to dictate how an argument is constructed—for instance, arguing based on expert authority versus potential consequences. Without this control, models frequently produce illogical leaps or factual hallucinations. ArgU addresses this by using "argument templates" and control codes to dictate both the stance (pro/con) and the logical structure, known as argument schemes.

Methodology: The ArgU Framework

The ArgU system employs a multi-stage approach involving data curation and a novel dual-model architecture.

1. Argument Understanding

Before generation, the system must understand existing arguments. We developed a RoBERTa-based tagger, ArgSpan, to identify factual spans in text, and ArgSpanScheme, a classification head that identifies which of six Walton’s argument schemes (e.g., Source Authority, From Consequence, Rule or Principle) the text follows.

2. ArgU-Dual Generation

Unlike standard "one-shot" generators, ArgU-Dual utilizes a two-step process:

  • Step 1: Template Generation. Given a set of facts and control codes (stance + scheme), the model generates an argument template—a logical skeleton with placeholders (e.g., <VAR_1> is known to <VAR_2>).
  • Step 2: Content Filling. The model fills this template with specific real-world concepts to produce a final, coherent argument.
The ArgU framework: generating arguments from factual variables, stance, and scheme using intermediate templates.

Conditioning the model on special tokens like <pro>, <con>, and scheme-specific tags allows for the generation of multiple distinct arguments from the same input facts by simply swapping control codes.

The ArgU Corpus

To train and evaluate these models, we introduced the ArgU Corpus, a significant new resource for the NLP community. It contains 69,428 arguments spanning six controversial topics: abortion (AB), minimum wage (MW), nuclear energy (NE), gun control (GC), death penalty (DP), and school uniforms (SU).

Dataset topics and cross-validation splits used for training and evaluation.

The dataset was created using a "silver-standard" approach, where a small human-annotated set (1,153 examples) was used to train the ArgSpanScheme model, which then automatically annotated a much larger parallel corpus. This represents the largest publicly available corpus for identifying argument schemes to date.

Key Insights and Results

Our evaluation revealed several critical findings regarding the intersection of logic and neural generation:

  • Logic as Scaffolding: Generating a template first acts as a logical constraint. By separating structure from content, the model significantly reduces illogical "hallucinations."
  • Scheme Invariance: Argument schemes are largely topic-independent. An argument "From Consequence" follows the same logical flow across different subjects. Masking specific facts during training allows the model to learn these abstract reasoning patterns.
  • Performance: ArgU-Dual outperformed single-step "Mono" versions in automatic metrics, achieving a BLEU score of 0.158 and a Rouge-L of 0.381. Human evaluations (on a 1-5 scale) rated the model highly for Fluency (4.86) and Logical Coherence (4.06).
  • Fact Faithfulness: The model maintained high semantic similarity (~0.64) between input facts and output text, demonstrating successful knowledge incorporation.

Limitations and Future Directions

While ArgU represents a significant step forward, several challenges remain. The model occasionally struggles with linguistic nuances, such as correctly inverting logic when an input variable contains negative verbs (e.g., "stop"). Additionally, "template drifting" can occur in the second generation step, where the decoder modifies the template, potentially altering the intended logical scheme. Future work will focus on expanding the system's zero-shot capabilities to new topics and extending the generator to handle longer-form rhetorical synthesis beyond the current 50-token limit.

Summary updated

Abstract

Effective argumentation is essential towards a purposeful conversation with a satisfactory outcome. For example, persuading someone to reconsider smoking might involve empathetic, well founded arguments based on facts and expert opinions about its ill-effects and the consequences on one’s family. However, the automatic generation of high-quality factual arguments can be challenging. Addressing existing controllability issues can make the recent advances in computational models for argument generation a potential solution. In this paper, we introduce ArgU: a neural argument generator capable of producing factual arguments from input facts and real-world concepts that can be explicitly controlled for stance and argument structure using Walton’s argument scheme-based control codes. Unfortunately, computational argument generation is a relatively new field and lacks datasets conducive to training. Hence, we have compiled and released an annotated corpora of 69,428 arguments spanning six topics and six argument schemes, making it the largest publicly available corpus for identifying argument schemes; the paper details our annotation and dataset creation framework. We further experiment with an argument generation strategy that establishes an inference strategy by generating an “argument template” before actual argument generation. Our results demonstrate that it is possible to automatically generate diverse arguments exhibiting different inference patterns for the same set of facts by using control codes based on argument schemes and stance.
BibTeX
@inproceedings{saha2023argu,
  author = {Sougata Saha and Rohini K. Srihari},
  title = {{ArgU: A Controllable Factual Argument Generator}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=yPQvXlZjR4},
}
Figure