Abstract
To combat the "infodemic" of COVID-19 misinformation, researchers have developed a clustering-based method to select the most effective subset of search engines for data fusion. By grouping retrieval systems based on the similarity of their results and picking the top performers from diverse clusters, they achieved a 31% improvement in retrieval precision over the best individual system.
Introduction
During the COVID-19 pandemic, the web was flooded with conflicting information, making it difficult for search engines to prioritize "relevant, credible, and correct" documents. A common technique to improve search quality is data fusion—combining the results of multiple retrieval systems (the "wisdom of the crowd").
However, data fusion faces a "Goldilocks" problem: using too few systems limits performance, but using too many (e.g., 50+) becomes computationally inefficient and can actually degrade quality if the systems are too similar or poor-performing. The authors address the challenge of subset selection: how do we choose a small, optimal group of search systems from a large pool to maximize both accuracy and diversity?
Methodology
The authors propose a Clustering-Based Subset Selection approach. Instead of just picking the systems with the highest individual scores, they use a two-step process:
- Clustering Systems: They treat each retrieval system as a data point. Using the Euclidean distance between the scores assigned to documents, they apply K-means clustering to group systems that produce similar results.
- Representative Selection: They select the best-performing system (based on Mean Average Precision) from each cluster. This ensures that the final subset isn't just a collection of "clones" that make the same mistakes, but a diverse team of high-performers.
Datasets and Evaluation
The study utilized the 2020 TREC (Text REtrieval Conference) Health Misinformation Track dataset, which includes 51 "runs" (sets of search results) and 50 queries focused on consumer health searches related to COVID-19. Documents were judged not just on relevance, but also on correctness and credibility. The authors emphasize the Convex Aggregation Measure (CAM), which balances these three dimensions.
Key Results and Insights
The "aha moment" of this research is that diversity is as important as accuracy. Picking the top 10 systems based on individual performance (a method called Top_J) is often sub-optimal because similar algorithms likely share the same blind spots. Clustering forces the selection of systems that "see" the document collection differently.
As shown in the baseline performance of individual systems above, the best single run achieved a MAP of 0.3832 and a CAM of 0.5883. By using their clustering method to pick just 17 systems (one-third of the total), the researchers achieved significant gains:
- Superiority over Single Systems: When fusing 17 selected runs using a linear combination, the method outperformed the best individual component system by 31.42% in MAP and 21.72% in CAM.
- Beating the State-of-the-Art: The clustering methods (C1 and C2) consistently outperformed the previous state-of-the-art subset selection method (Top_J) by a clear margin (often 5-8% improvement).
- Efficiency: The research shows that you don't need 50 systems to get the best results; a diverse subset can provide massive performance gains with lower computational overhead.
Limitations and Future Work
While the results are promising, the best performance came from the "Linear Combination" method, which requires a training set with human-labeled relevance judgments. In a rapidly evolving crisis, these labels might not be available quickly enough. Future work will focus on fully unsupervised selection methods and dynamically determining the optimal number of clusters based on the specific set of available systems.
Summary updated