TL;DR
The COVID-19 pandemic generated an overwhelming volume of scientific literature, making it difficult for experts to sift through and identify high-quality, relevant research. AMICA is a novel search engine that addresses this by not only matching keywords but also ranking papers based on their argumentative content, using argument mining techniques. This approach has shown promising results in aligning with expert opinions and established quality criteria.
What Problem Does This Paper Address?
The sheer volume of scientific publications, particularly preprints, during the COVID-19 pandemic created a significant challenge for researchers. Domain experts struggled to keep pace with the influx of information, filter out low-quality papers, and identify the most relevant and impactful research in a timely manner. Traditional keyword-based search engines, while useful, don't inherently distinguish between well-supported arguments and less substantial content, leading to an information overload. The paper aims to bridge this gap by developing a system that prioritizes papers with strong argumentative structures.
Methods
AMICA employs a two-pronged approach to ranking scientific literature:
- Argument Mining (AM) with MARGOT: The core of AMICA's novelty lies in its use of argument mining. Specifically, it leverages MARGOT, a freely available system that identifies "claims" (debatable statements) and "evidence" (supporting premises) within text. MARGOT assigns a "claim score" (CS) and an "evidence score" (ES) to each sentence. AMICA then calculates an "argumentative score" (AS) for each document by averaging the maximum of CS and ES across all its sentences. This score reflects the document's overall argumentative richness.
- Hybrid Scoring Function: AMICA combines the argumentative score (AS) with a traditional information retrieval score (s(q, dj)). The traditional score is based on the similarity between the user's query and the document's content, typically using bag-of-words representations. The final AMICA score (sa) is the product of these two scores:
sa(q, dj) = s(q, dj) * AS(dj). This ensures that papers are both relevant to the query and rich in argumentative content.
Datasets
- COVID-19 Scientific Literature Corpus: AMICA utilizes a continuously updated dataset of COVID-19 related scientific papers. These papers are sourced from open-access archives like arXiv, bioRxiv, medRxiv, and Research Square. The system processes over 15,000 papers, which have been annotated by MARGOT for argument detection.
- Cochrane Systematic Review: For evaluation, the authors used a Cochrane Systematic Review on rapid, point-of-care and molecular-based tests for COVID-19. This review provided a curated list of 40 papers, categorized as either "included" or "excluded" based on rigorous eligibility criteria.
Key Insights and Ideas
- Argumentative Content as a Quality Indicator: The paper's central hypothesis is that argumentative content (claims and evidence) can serve as a proxy for paper quality and relevance, helping researchers prioritize their reading.
- Beyond Keyword Matching: AMICA moves beyond simple keyword matching by incorporating a deeper understanding of the text's structure and persuasive elements.
- MARGOT's Generalizability: The study highlights MARGOT's ability to generalize across different document genres and styles, making it suitable for analyzing scientific literature.
- Correlation with Expert Opinion: The experimental evaluation demonstrates a significant correlation between AMICA's ranking and the judgment of domain experts, suggesting its practical utility.
Key Results
- Alignment with Cochrane Reviews: In a case study involving 40 papers from a Cochrane Systematic Review, AMICA's analysis showed a clear distinction between papers included and excluded from the review. Papers included in the review generally had higher argumentative ratios (AR) and argumentative scores (AS), indicating that well-argued papers are more likely to be considered high-quality by systematic reviewers.
- Correlation with Expert Scores: When comparing AMICA's argumentative score (AS) and argument ratio (AR) with scores provided by four domain experts, the authors found moderate-to-strong Spearman's rank correlation coefficients (ρ = 0.463 for AS, ρ = 0.526 for AR). This suggests that AMICA's argumentative metrics align well with expert assessments of paper quality.
Limitations and Future Work
- Limitations: The authors acknowledge that more sophisticated scoring functions could be conceived. The current hybrid approach is noted as being simple and efficient, but its effectiveness for all scenarios is not guaranteed.
- Future Work: The authors plan to extend AMICA's approach to other domains within biomedical literature. They also intend to compare AMICA's ranking performance against other existing search engines.
Summary updated