TL;DR
This paper introduces SciConceptMiner, a novel system that automatically discovers emerging scientific concepts from vast amounts of academic literature and structured knowledge bases. By combining a self-supervised BERT-based model with web search for validation, it achieves highly accurate concept identification, powering semantic search in systems like Microsoft Academic Graph.
What Problem Does This Paper Address?
The scientific landscape is exploding with new information, with millions of academic articles published monthly. This rapid growth makes it incredibly challenging to keep track of emerging concepts and understand the evolving structure of scientific knowledge. Traditional topic modeling methods, like Latent Dirichlet Allocation (LDA), struggle with scalability, both in terms of the sheer volume of documents and the number of concepts they can identify. Furthermore, LDA outputs often require manual annotation to become human-readable, a costly and time-consuming process. There's a pressing need for an automated system that can efficiently and accurately discover, cluster, and validate new scientific concepts from diverse sources.
Methods
SciConceptMiner employs a two-stage approach:
- Concept Candidate Discovery:
- Leveraging Existing Knowledge: The system first taps into semi-structured knowledge sources like Wikipedia and the Unified Medical Language System (UMLS). These sources provide a rich, albeit sometimes noisy, list of potential concepts.
- Self-Supervised BERT Model: A BERT-based sequence labeling model is trained using self-supervised data derived from these existing concepts. This model learns to identify candidate concept phrases within academic documents by predicting whether a word or phrase represents a scientific concept. This approach is inspired by Named Entity Recognition (NER) techniques but is adapted for the specific task of scientific concept extraction.
- Synonym Detection & Concept Selection:
- Web Search API: Candidate concepts identified in the first stage are then fed into a web search API (Bing Web Search).
- Clustering and Validation: The system analyzes the search results, focusing on URL relevance, domain reputation, and the consistency of mentions across top search results. This allows it to cluster synonyms, identify legitimate concepts, and discard irrelevant or low-quality terms. A curated list of allowed and blocked URL domains further refines this selection process.
Datasets
- Training Data: The system is trained on a corpus derived from Microsoft Academic Graph (MAG), specifically from Computer Science (CS) and Medicine domains. This includes articles, sentences, and tokens, with a focus on identifying concept tokens.
- External Knowledge Sources: Wikipedia and the Unified Medical Language System (UMLS) are used as initial sources for concept candidates and for generating self-supervised training data.
- Inference Data: The trained models are applied to larger inference corpora from CS and Medicine domains within MAG.
Key Insights and Ideas
- Self-Supervision is Key: The ability to leverage existing, albeit imperfect, knowledge bases (Wikipedia, UMLS) to train a model for discovering new concepts is a powerful idea. This avoids the need for large, manually labeled datasets of novel scientific terms.
- BERT for Scientific Language: Adapting BERT, a powerful language model, for the specific nuances of scientific text and concept extraction proves highly effective.
- Web Search as a Validator: Using a web search API not just for information retrieval but as a mechanism for concept validation and synonym clustering is an innovative way to leverage the vastness of the web for scientific knowledge discovery.
- Domain-Specific Models: The paper highlights that models trained on domain-specific corpora (e.g., CS vs. Medicine) perform significantly better within their respective domains, suggesting a need for specialized concept extractors.
Key Results
- High Accuracy: SciConceptMiner achieves a consistently high accuracy of 94-95% in identifying scientific concepts.
- Large-Scale Discovery: The system has successfully identified over 740,000 scientific concepts, which are now integrated into the Microsoft Academic Graph.
- Efficiency: The two-stage approach, particularly the web search validation, effectively filters out a large majority (92-97%) of noisy terms, keeping only high-quality concepts and synonyms.
- Domain Performance: F1 scores for in-domain testing are high (e.g., 0.942 for CS-Model on CS-Test), while out-of-domain performance drops (e.g., 0.682 for CS-Model on Medicine-Test), reinforcing the benefit of domain-specific models.
Limitations and Future Work
- Domain Specificity: The current models are trained on CS and Medicine. The authors acknowledge that continued training on other discipline corpora and exploring more effective concept extractor learning models are ongoing efforts.
- Coverage Estimation: While the system is evaluated for accuracy, estimating full coverage is challenging. The authors estimate that they may have missed around 100K concepts and 200K synonyms from the inference results of their models.
- Concept Age: Concepts from UMLS tend to be older and have a longer lifespan, while automatically extracted concepts (AutoCS, AutoMed) are generally newer, aligning with the goal of discovering emerging trends. Future work could explore how to better capture and represent these emerging concepts.
Summary updated