DFPE: A Diverse Fingerprint Ensemble for Enhancing LLM Performance

Seffi Cohen, Niv Goldshlager, Nurit Cohen-Inger, Bracha Shapira, Lior Rokach

TL;DR

DFPE (Diverse Fingerprint Ensemble) is a training-free ensemble method that boosts LLM performance by selecting a diverse, high-performing subset of models tailored to specific subjects. By clustering models based on their response "fingerprints" and applying adaptive weighting, it achieves a 3% overall and 5% discipline-level accuracy gain on the MMLU benchmark without the need for expensive fine-tuning.

What Problem Does This Paper Address?

Large Language Models (LLMs) are impressive generalists, but they rarely excel uniformly across all domains. A model that is a "savant" in quantum physics might be mediocre at contract law. While ensembling multiple models is a proven way to mitigate these individual weaknesses, standard approaches like majority voting often fall short.

The core challenges in LLM ensembling are redundancy and competence. If you ensemble five models that all share the same training data and biases, they will likely fail on the same difficult questions, providing no net benefit. Conversely, including weak models can "pollute" the ensemble's decision. The authors seek a way to dynamically select a diverse yet expert group of models for any given subject without the massive computational overhead of retraining or complex routing architectures.

Methods

The DFPE pipeline operates in four distinct, training-free stages:

  1. Model Fingerprinting & Clustering: For a specific subject (e.g., "Abstract Algebra"), each model's responses on a small validation set are converted into embedding vectors using a sentence transformer. These "fingerprints" represent the model's unique problem-solving pattern. The authors then use DBSCAN to cluster models with similar fingerprints, effectively grouping models that "think" alike.
  2. Quantile-based Selection: To ensure a baseline of competence, the authors calculate a subject-specific accuracy threshold. Models falling below a certain quantile (e.g., the bottom 5%) are discarded.
  3. Diversity Preservation: From each remaining cluster, only the single highest-performing model is selected as a representative. This systematically filters out redundant models while maintaining a variety of solution strategies.
  4. Adaptive Weighting: The selected models are assigned weights based on an exponential scaling of their validation accuracy ($w = \exp(\gamma \cdot \text{accuracy})$). This gives significantly more influence to the "subject matter experts" while still allowing diverse perspectives to contribute to the final weighted vote.

The entire process is illustrated in the following figure: DFPE Pipeline Overview

DFPE Pipeline Overview

Datasets

The method was evaluated using the Massive Multitask Language Understanding (MMLU) benchmark, which consists of multiple-choice questions across 57 subjects.

The "model pool" consisted of 10 diverse, open-source LLMs ranging from 3B to 9B parameters, including:

  • GLM-4-9B
  • Qwen2.5 (7B & 3B)
  • Llama-3.1-8B
  • Phi-3.5-mini
  • Mistral v0.3
  • Gemma-2-9B, Apollo2-7B, Starling-LM-7B, and Yi-1.5-6B.

Key Insights and Ideas

  • Diversity > Raw Accuracy: One of the most important findings is that a smaller ensemble of diverse models often outperforms a larger ensemble of highly accurate but similar models. By picking the best representative from each cluster, DFPE avoids the "echo chamber" effect.
  • Fingerprints as Proxies for Logic: Using response embeddings as "fingerprints" allows the system to capture subtle behavioral similarities between models that simple accuracy metrics miss.
  • Subject-Level Granularity: The ensemble composition is not static. DFPE re-evaluates which models are "experts" and which are "redundant" for every single subject in the MMLU, allowing for fine-grained adaptation.
  • Efficiency via Output-Level Fusion: Because DFPE operates on the final predicted choices rather than internal tokens or hidden states, it is highly scalable and avoids the complex vocabulary-matching issues prevalent in other ensemble methods.

Key Results

DFPE demonstrated consistent improvements across the board:

  • Overall Accuracy: Achieved 73.5%, outperforming the best single model (GLM-4 at 70.8%) by nearly 3%.
  • Discipline-Level Accuracy: Achieved 74.0%, a 5% absolute improvement over the best single model's discipline-balanced score.
  • Broad-Spectrum Gains: As shown in Figure 2, DFPE matched or exceeded baseline methods (like Majority Voting) across a wide range of disciplines, from STEM to Humanities.
Figure 2: Average Accuracy by Discipline. DFPE consistently outperforms the compared methods across a wide range of disciplines, highlighting broad-spectrum gains.

Sensitivity analysis (Figure 3) showed that the method is robust to hyperparameter choices, with optimal performance found at a low quantile threshold (0.05) and an accuracy scaling factor ($\gamma$) of 5.0. Sensitivity Analysis

Sensitivity Analysis

Limitations and Future Work

  • Inference Overhead: The primary drawback is the need to run multiple models to generate a single answer, which increases computational cost and latency.
  • Validation Data Dependency: The method requires a small amount of labeled validation data per subject to generate fingerprints and weights.
  • Task Constraints: The current implementation is optimized for multiple-choice questions. Extending this "fingerprinting" logic to open-ended generative tasks (like code generation or creative writing) remains an open research direction.
  • Model Pool Homogeneity: The authors note that if the available pool of models is too similar (e.g., all fine-tuned versions of the same base model), the benefits of clustering and diversity preservation diminish.

Summary updated

Abstract

Large Language Models (LLMs) have shown remarkable capabilities across various natural language processing tasks but often struggle to excel uniformly in diverse or complex domains. We propose a novel ensemble method - Diverse Fingerprint Ensemble (DFPE), which leverages the complementary strengths of multiple LLMs to achieve more robust performance. Our approach involves: (1) clustering models based on response "fingerprints" patterns, (2) applying a quantile-based filtering mechanism to remove underperforming models at a per-subject level, and (3) assigning adaptive weights to remaining models based on their subject-wise validation accuracy. In experiments on the Massive Multitask Language Understanding (MMLU) benchmark, DFPE outperforms the best single model by 3% overall accuracy and 5% in discipline-level accuracy. This method increases the robustness and generalization of LLMs and underscores how model selection, diversity preservation, and performance-driven weighting can effectively address challenging, multi-faceted language understanding tasks.
BibTeX
@inproceedings{cohen2026dfpe,
  author = {Seffi Cohen and Niv Goldshlager and Nurit Cohen-Inger and Bracha Shapira and Lior Rokach},
  title = {{DFPE: A Diverse Fingerprint Ensemble for Enhancing LLM Performance}},
  booktitle = {EACL (Findings)},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.282},
}
Figure