Webis at CQs-Gen 2025: Prompting and Reranking for Critical Questions

Midhun Kanadan, Johannes Kiesel, Maximilian Heinrich, Benno SteinACL 2025

TL;DR

This paper presents a two-stage pipeline for automatically generating critical questions that challenge arguments. By combining diverse prompting strategies for Large Language Models (LLMs) with a fine-tuned reranking classifier, the system achieved a strong 5th place in the Critical Question Generation shared task, demonstrating the effectiveness of this approach for improving question quality.

What Problem Does This Paper Address?

The paper addresses the challenge of automatically generating "critical questions" (CQs) that probe the underlying assumptions, reasoning, evidence, or implications of an argument. Unlike counterarguments, CQs don't need to be factually incompatible with the original statement; instead, they aim to expose an argument's "blind spots." While LLMs are fluent generators, they often struggle with hallucinations, outdated knowledge, or superficial reasoning, making them unreliable for producing factual counterarguments. The Critical Question Generation (CQG) shared task aims to automate the creation of these challenging questions, which are crucial for critical thinking and argumentation analysis. Existing datasets for training robust CQ generation models are limited in size and coverage, posing a significant hurdle.

Methods

The team "Webis" developed a two-stage pipeline:

  1. Prompting for Critical Questions: This stage involves using Large Language Models (LLMs) to generate candidate critical questions. They explored various prompting strategies:
  2. Zero-shot prompting: Directly asking the LLM to generate CQs.
  3. Guideline-based prompting: Incorporating definitions of CQs and their purpose.
  4. Chain-of-thought prompting: Encouraging step-by-step reasoning before generation.
  5. Few-shot prompting: Providing examples of interventions, CQs, definitions, and self-assessment checks to guide the LLM. They tested a range of open-weight LLMs (Gemma 3, Gemma 2, LLaMA 3.2, Mistral, Phi-4, Qwen 2.5) and a closed-source model (GPT-4o-mini). An experimental approach also integrated argumentation schemes from Walton et al. into the prompts, but this did not yield the best results.

  6. Reranking Critical Questions: To select the most useful questions from the LLM outputs, they employed a reranking stage using a fine-tuned ModernBERT classifier.

  7. Model Selection: ModernBERT was chosen for its ability to handle longer inputs, which is beneficial for processing argumentative interventions.
  8. Fine-tuning: Two variants of ModernBERT were fine-tuned: one on the official validation dataset and another on an augmented dataset of approximately 67.8k CQs generated by LLMs using diverse prompting strategies. This augmented model was further trained on questions automatically labeled as "Useful," "Unhelpful," or "Invalid" based on semantic similarity to reference questions.
  9. Classification: The classifier was trained for binary classification (Useful vs. Non-Useful).
  10. Pipeline Integration: The final system combined outputs from multiple LLMs, and the ModernBERT classifier scored each generated question for its predicted usefulness. The top three questions were then selected.

Datasets

The primary dataset used was the one provided by the ArgMining 2025 Critical Question Generation shared task organizers. This dataset includes argumentative interventions, reference questions labeled as "Useful," "Unhelpful," or "Invalid," and argumentation scheme annotations. The authors also created an augmented dataset for fine-tuning their reranking model, consisting of approximately 67.8k CQs generated by LLMs.

Key Insights and Ideas

  • Two-stage pipeline is crucial: Combining LLM generation with a downstream reranking mechanism significantly improves the quality of critical questions compared to generation alone.
  • Reranking is key to quality: The reranking stage, powered by a fine-tuned classifier, is essential for filtering out less useful questions and selecting the most impactful ones.
  • Augmented training data boosts performance: Fine-tuning the reranking model on a larger, augmented dataset of generated questions led to better generalization and improved performance.
  • Prompt diversity matters: Using a variety of prompting strategies (especially few-shot and guideline-based) helps generate more diverse and relevant candidate questions.
  • Lightweight LLMs can perform well: Even relatively smaller open-weight LLMs, when combined with effective prompting and reranking, can achieve competitive results in complex generative tasks.
  • Context is king (but can be limiting): While longer interventions provide more context for LLMs, very short or single-sentence interventions can lead to generic questions due to limited argumentative material.
  • Evaluation limitations: Similarity-based evaluation metrics might undervalue useful questions that differ lexically from reference examples.

Key Results

The best-performing configuration, "Reranker-Augmented," achieved a test score of 0.57, ranking 5th in the shared task. This configuration used multiple LLMs for generation and a ModernBERT reranker trained on both the validation data and the augmented dataset. A "Reranker-Base" configuration, using the same generation setup but a reranker trained only on the validation set, achieved a lower score of 0.48, highlighting the benefit of the augmented training data. Submissions with reranking consistently outperformed baseline submissions without reranking across all metrics.

Limitations and Future Work

  • Short interventions: The paper notes that short or single-sentence interventions can lead to overly generic or unhelpful critical questions because the LLMs have limited argumentative context.
  • Lexical bias in evaluation: The reliance on similarity-based evaluation can potentially undervalue useful questions that are lexically different from the reference examples.
  • Future Work: The authors suggest exploring agent-based iterative generation strategies where a CQ generation model and a feedback module interact to refine questions over multiple rounds. They also propose using instruction-tuned models or reinforcement learning to explicitly optimize for question usefulness and specificity.

Summary updated

BibTeX
@inproceedings{kanadan2025webis,
  author = {Midhun Kanadan and Johannes Kiesel and Maximilian Heinrich and Benno Stein},
  title = {{Webis at CQs-Gen 2025: Prompting and Reranking for Critical Questions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure