The Manchester Bees at PerAnsSumm 2025: Iterative Self-Prompting with Claude and o1 for Perspective-aware Healthcare Answer Summarisation

Pablo Romero, Libo Ren, Lifeng Han, Goran Nenadić2025 · DOI

Abstract

This system report presents an innovative approach to the PerAnsSumm2025 shared task at the Workshop CL4Health, addressing the critical challenges of perspective-aware healthcare answer summarization. Our method, Iterative Self-Prompting (ISP) with Claude and o1, introduces a novel framework that leverages large language models' ability to iteratively refine their own instructions, achieving competitive results without traditional model training. Despite utilising only API calls rather than computational-intensive training, our system "The Manchester Bees" secured 15th place among 23 leaderboard systems overall, while demonstrating exceptional performance in key metrics—ranking 6th in Strict-matching-F1 for span identification (Task A) and achieving the highest Factuality score for summary generation (Task B). Notably, our approach achieved state-of-the-art results in specific metrics, including the highest Strict-matching precision (0.2267) for Task A and AlignScore (0.5888) for Task B. This performance, accomplished with minimal computational resources and development time measured in hours rather than weeks, demonstrates the potential of ISP to democratise access to advanced NLP capabilities in healthcare applications.

Introduction

Online health forums offer a wealth of peer support and lived experience, but they are notoriously unstructured. The PerAnsSumm 2025 shared task challenges researchers to generate "perspective-aware" summaries of these forum posts, categorizing information into five domains: information, cause, suggestion, experience, and question.

Traditional approaches to this problem usually involve fine-tuning models like BERT or T5, which requires significant labeled data, computational power, and development time. The "Manchester Bees" team sought to bridge the gap between high-performance requirements and the need for rapid, low-resource deployment in healthcare settings using a novel prompting strategy.

Methodology: Iterative Self-Prompting (ISP)

The core innovation of this work is Iterative Self-Prompting (ISP). Instead of humans painstakingly engineering the "perfect" prompt, the authors leveraged the meta-cognitive abilities of LLMs (specifically Claude and o1) to refine their own instructions through a feedback loop.

The Iterative Self-Prompting (ISP) workflow consisting of four stages: First Prompt, Test, Refine, and Result.

The ISP workflow follows a four-stage process:

  1. First Prompt: An LLM is given a detailed task description and asked to write a prompt that another model could use to solve the task.
  2. Test: This generated prompt is tested on a small set of training data.
  3. Refine: The model is shown its previous prompt, the output it generated, and the ground truth. The model then performs a detailed analysis of its errors and updates the prompt to handle those specific edge cases.
  4. Result: This cycle repeats until the prompt is highly optimized.

To handle the technical constraints of long forum posts, the team split the test set into manageable chunks of five questions each, ensuring the models maintained sufficient context without hitting token limits.

Key Results and Efficiency

Despite having zero traditional training and a development window of only 3–4 hours, the system delivered impressive results across both competition tasks.

Performance Highlights

  • Factuality: Version 3 of the system achieved the #1 rank in Factuality (measured by AlignScore) for Task B, reaching a score of 0.5888.
  • Precision: In Task A (Span Identification), the system ranked 1st in Strict-matching Precision (0.2267) and 6th in F1 score among 23 competing systems.
  • Overall Standing: The team secured 15th place overall on the leaderboard, outperforming many systems that utilized far more computational resources.

Comparison of Approaches

The success of ISP suggests that advanced NLP capabilities can be "democratized." Small teams or clinics without massive server farms can achieve competitive results using only API calls and a few hours of smart prompting.

Comparison of resource requirements and development time between traditional fine-tuning and the Iterative Self-Prompting (ISP) approach.

Discussion and Insights

  • Autonomous Pattern Recognition: The model autonomously realized that certain categories (like "Question" or "Experience") are often naturally empty in high-quality answers and modified its own prompt to handle these "empty categories" without human intervention.
  • The "Calibration Down" Phenomenon: The authors observed that LLMs were occasionally too good. They had to "calibrate down" the model's responses to better align with the specific formatting and quality patterns expected by the competition's evaluation metrics.

Conclusion

The Manchester Bees' performance demonstrates that Iterative Self-Prompting is a viable, high-efficiency alternative to traditional fine-tuning for specialized medical NLP tasks. Future work will look into more sophisticated, automated chunking strategies to overcome context window constraints and direct comparisons against fine-tuned encoder-decoder models like T5.

Summary updated

Abstract

This system report presents an innovative approach to the PerAnsSumm2025 shared task at the Workshop CL4Health, addressing the critical challenges of perspective-aware healthcare answer summarization.Our method, Iterative Self-Prompting (ISP) with Claude and o1, introduces a novel framework that leverages large language models' ability to iteratively refine their own instructions, achieving competitive results without traditional model training.Despite utilising only API calls rather than computational-intensive training, our system "The Manchester Bees" secured 15th place among 23 leader board systems overall, while demonstrating exceptional performance in key metrics -ranking 6th in Strict-matching-F1 for span identification (Task A) and achieving the highest Factuality score for summary generation (Task B).Notably, our approach achieved state-of-the-art results in specific metrics, including the highest Strict-matching precision (0.2267) for Task A and AlignScore (0.5888) for Task B. This performance, accomplished with minimal computational resources and development time measured in hours rather than weeks, demonstrates the potential of ISP to democratise access to advanced NLP capabilities in healthcare applications.Our complete implementation is available as an open-source project on https://github.com/ pabloRom2004/-PerAnsSumm-2025
BibTeX
@misc{romero2025manchester,
  author = {Pablo Romero and Libo Ren and Lifeng Han and Goran Nenadić},
  title = {{The Manchester Bees at PerAnsSumm 2025: Iterative Self-Prompting with Claude and o1 for Perspective-aware Healthcare Answer Summarisation}},
  year = {2025},
  doi = {10.18653/v1/2025.cl4health-1.30},
}
Figure