RECSIP: REpeated Clustering of Scores Improving the Precision

André Schamschurko, Nenad Petrović, Alois KnollLecture Notes in Networks and Systems 2025 · DOI · OpenAlex

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, yet their deployment in high-stakes environments—such as medicine, law, or industrial robotics—remains limited by their inherent stochasticity. LLMs are prone to hallucinations and can provide inconsistent answers to the same prompt. Traditional methods to mitigate these issues, such as self-consistency (querying a single model multiple times) or multi-agent debates (using one LLM to judge others), often fall short. Self-consistency fails if the underlying model has a fundamental knowledge gap, while LLM-as-a-judge frameworks often inherit the biases and errors of the evaluator model.

To address these challenges, we present RECSIP (REpeated Clustering of Scores Improving the Precision). RECSIP is a novel framework designed to harness the "wisdom of the crowd" by querying multiple diverse model architectures in parallel and employing an iterative, objective clustering mechanism to reach a reliable consensus.

The RECSIP Framework

RECSIP replaces subjective human or LLM evaluation with a multi-stage pipeline rooted in objective NLP similarity metrics. The process follows five distinct stages:

1. Parallel Answering

A single query is dispatched simultaneously to $n$ different LLMs. By utilizing models from different families (e.g., OpenAI’s GPT, Anthropic’s Claude, and Google’s Gemini), the system benefits from architectural diversity, reducing the likelihood of shared training biases or synchronized errors.

2. Scoring and Clustering

Rather than relying on an LLM to "rate" the quality of responses, RECSIP calculates similarity scores between all generated answers using objective metrics: ROUGE, METEOR, and Cross-Encoders (specifically stsb-distilroberta-base). These scores are used to group similar responses into clusters. This approach avoids the circular logic of using a fallible LLM to verify another LLM.

3. The Callback Loop

If the initial generation results in multiple clusters (indicating a lack of consensus), the framework initiates a "callback." It generates a new prompt that presents the original question as a multiple-choice task. Each option in this new prompt represents one of the previously identified clusters, effectively summarizing the collective "opinions" of the ensemble.

4. Iterative Refinement

The models are then asked to vote on these options. This process repeats iteratively until the models converge on a single cluster or the number of clusters stops decreasing. This allows models to reconsider their reasoning in light of alternative perspectives provided by their peers.

5. The Safety Valve

A critical feature of RECSIP is its ability to signal uncertainty. If the iterative process fails to reach a consensus, the system returns a "Disagreement Response." In high-risk applications, knowing that the system is uncertain is far more valuable than receiving a confident but incorrect answer.

Experimental Evaluation

Dataset: MMLU-Pro

The framework was evaluated using MMLU-Pro, a rigorous extension of the Massive Multitask Language Understanding benchmark. MMLU-Pro is significantly more challenging than the original MMLU, increasing the number of response options from 4 to 10 and focusing on reasoning-heavy tasks across various academic and professional subjects.

Model Ensemble

The reference implementation utilized three state-of-the-art models:

  • GPT-4o
  • Claude 3.5 Sonnet
  • Gemini 1.5 Pro

Results and Discussion

RECSIP demonstrated a dominant performance across the MMLU-Pro benchmark, consistently outperforming individual models and standard ensemble techniques.

  • Overall Accuracy: RECSIP achieved an overall score of 0.835, representing a 5.8 percentage point improvement over the best-performing individual model in the ensemble (Claude 3.5 Sonnet at 0.776).
  • Category Performance: The framework outperformed the leading individual model in every single category of the benchmark. Notable gains were observed in Engineering (a 10.3% increase) and Computer Science (a 2.0% increase).
  • Precision and Reliability: By identifying "disunity" among models, RECSIP successfully filtered out incorrect responses in complex fields like Law and Engineering, significantly boosting the precision of the final output.

The success of RECSIP highlights two key insights: first, that architectural diversity is essential for error correction; and second, that objective similarity metrics provide a more stable foundation for consensus than LLM-based judges.

Limitations and Future Work

Despite its performance gains, RECSIP introduces certain trade-offs:

  • Computational Cost: Querying multiple high-tier models and running cross-encoder scoring increases latency and API costs compared to single-model inference.
  • The Unanimous Error Problem: The primary failure mode occurs when all models in the ensemble are initially incorrect. In the Biology category, for instance, 32 out of 45 errors were caused by a unanimous initial mistake. If the entire "crowd" is wrong, the clustering mechanism cannot recover the truth.

Future research will explore weighted voting, where models are assigned different influence levels based on their historical performance in specific subjects (e.g., prioritizing Gemini for Physics). Additionally, the authors plan to extend the framework to non-natural language tasks, such as code generation and formal logic, to test its versatility across different data structures.

Summary updated

BibTeX
@inproceedings{schamschurko2025recsip,
  author = {André Schamschurko and Nenad Petrović and Alois Knoll},
  title = {{RECSIP: REpeated Clustering of Scores Improving the Precision}},
  booktitle = {Lecture Notes in Networks and Systems},
  year = {2025},
  doi = {10.1007/978-3-032-00071-2\_7},
}

Loading related content...

Figure