Perspective-driven Preference Optimization with Entropy Maximization for Diverse Argument Generation

Yilin Cao, Ruike Zhang, Penghui Wei, Qingchao Kong, Wenji MaoEMNLP 2025

TL;DR

This paper introduces POEM, a novel framework for generating diverse arguments that represent multiple perspectives on a given claim and stance. By combining instruction fine-tuning, preference optimization, and entropy maximization, POEM significantly enhances perspective diversity while maintaining high quality and controllability, addressing a key limitation in current argument generation models.

What Problem Does This Paper Address?

Generating arguments that are not only coherent and relevant but also represent a variety of viewpoints is crucial for fostering balanced discourse and understanding complex, controversial topics. While large language models (LLMs) have shown impressive generative capabilities, they often struggle to produce diverse perspectives, tending to generate repetitive or semantically similar arguments. Existing methods have largely overlooked the explicit generation of multi-perspective arguments under quality constraints like claim-stance alignment. This gap limits the ability of AI systems to model nuanced human communication and can lead to biased outputs.

Methods

POEM tackles this challenge with a three-module framework:

  1. Instruction Fine-Tuning: This initial stage equips the LLM with the basic ability to generate arguments that are aligned with the given claim and stance. It implicitly learns from high-quality, multi-perspective argument sets.
  2. Perspective-driven Preference Optimization: This module explicitly enhances perspective diversity. It involves:
  3. Perspective Mining: Extracting salient perspectives from generated arguments using noun phrase parsing and Latent Dirichlet Allocation (LDA) for topic clustering.
  4. Diversity Measuring: Quantifying the perspective diversity of candidate argument sets using KL divergence between similarity scores to perspectives.
  5. Preference Data Construction: Creating a preference dataset by pairing highly diverse argument sets with less diverse ones.
  6. Preference Optimization: Fine-tuning the model using Direct Preference Optimization (DPO) to favor more diverse argument sets.
  7. Entropy Maximization: This module further promotes diversity by encouraging dispersed semantic representations of the generated arguments. It uses a semantic regularization loss based on the Coding-Length Function (CLF) to expand the representation space and reduce semantic redundancy.

Datasets

The authors evaluate their method on three datasets:

  • PERSPECTRUM: A benchmark dataset for stance-conditional argument generation, containing claims and diverse sentential arguments.
  • CMV (Change My View): A dataset derived from the Change My View subreddit, focusing on debate threads and typically containing more open-ended and subjective claims.
  • MIXED: A newly constructed dataset for out-of-domain evaluation, comprising claims from various political, social, and public health domains.

Key Insights and Ideas

  • Explicitly optimizing for perspective diversity is key: Simply relying on LLMs' inherent capabilities or basic prompting isn't enough. A dedicated mechanism to measure and reward diverse perspectives is crucial.
  • Semantic representation diversity matters: Encouraging arguments to have distinct semantic embeddings (via entropy maximization) helps prevent redundancy and broadens the range of covered perspectives.
  • A multi-stage approach is effective: Combining instruction fine-tuning for basic control, preference optimization for explicit diversity guidance, and entropy maximization for deeper semantic separation yields superior results.
  • Transferability across models and domains: POEM demonstrates strong generalization capabilities, performing well on different LLM architectures and out-of-domain datasets like CMV and MIXED.

Key Results

POEM consistently outperforms baselines in perspective diversity and semantic diversity across all tested datasets. For instance, on the PERSPECTRUM dataset, POEM (Meta-Llama-3-8B-Instruct) achieves a perspective diversity score of 8.46, significantly higher than baselines like GPT-4-turbo (8.32). Crucially, this improvement in diversity is achieved while maintaining competitive performance in claim relevance, stance alignment, and text quality (measured by perplexity). Human evaluations also confirm POEM's superiority in perspective diversity, with scores averaging 8.31 for the Meta-Llama-3-8B-Instruct variant on the CMV dataset, compared to GPT-4's 8.48. Ablation studies show that each component of POEM contributes significantly to its overall performance.

Limitations and Future Work

The authors acknowledge that their method is primarily designed for paragraph-level argument generation and may face challenges with longer, discourse-level arguments requiring complex logical structures and extended coherence. Additionally, the current work does not address cases with neutral or ambiguous stances, which remain an open challenge. Future work could explore handling stance ambiguity and extending the framework to more complex argumentative writing.

Summary updated

BibTeX
@inproceedings{cao2025perspective,
  author = {Yilin Cao and Ruike Zhang and Penghui Wei and Qingchao Kong and Wenji Mao},
  title = {{Perspective-driven Preference Optimization with Entropy Maximization for Diverse Argument Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure