TL;DR
Researchers often need to perform complex text mining and information extraction on the vast MEDLINE biomedical literature database. This paper presents open-source software tools and a relational database schema designed to efficiently load and query MEDLINE locally, overcoming limitations of the PubMed API and enabling more sophisticated analyses.
What Problem Does This Paper Address?
The MEDLINE database, containing over 12 million citations, is a treasure trove for biomedical research. However, accessing and analyzing this data programmatically can be challenging. The primary method for querying MEDLINE is through the PubMed web interface or its application programming interface (API). While useful, the PubMed API has limitations, including rate limits on queries and restrictions on complex, high-volume data extraction. Researchers interested in advanced text mining, natural language processing, or building custom applications often require direct, local access to the data. The National Library of Medicine (NLM) provides MEDLINE in XML format, but directly querying large XML files is inefficient. This paper addresses the need for a robust, local solution by developing tools to transform MEDLINE XML into a queryable relational database.
Methods
The core of this work involves developing software tools to parse MEDLINE's XML files and load them into a relational database. The authors designed a relational database schema based on the MEDLINE DTD, aiming for a de-normalized structure to optimize read-only query performance. They implemented three versions of parsing and loading software:
- Java MedlineParser: Developed at Berkeley, this tool uses the Java programming language and the SAX (Simple API for XML) parser for efficient XML processing. It leverages JDBC (Java Database Connectivity) for database interaction.
- Stanford Java Version: A modified version of the Berkeley Java tool, adapted to run at Stanford.
- Perl ParseMedline: Developed at Stanford, this tool uses the Perl programming language, the Perl SAX parser, and DBI (Database Interface) for database communication.
A key methodological choice was the use of the SAX parser over DOM (Document Object Model). SAX is preferred for large documents like MEDLINE because it processes data in a streaming fashion, avoiding the need to load the entire document into memory, thus being more resource-efficient and faster. The loading process offers two options: direct loading into the database or writing to intermediate flat files (e.g., comma-separated values) before loading.
Datasets
The primary dataset used is the MEDLINE database, provided by the National Library of Medicine (NLM) in XML format. The authors used data from 2002 and early 2003 for their implementations. They also mention using MeSH (Medical Subject Headings) XML files and LocusLink data for specific research applications. No new benchmarks were introduced; rather, the paper focuses on the performance of loading and querying the existing MEDLINE dataset.
Key Insights and Ideas
- Relational Databases are Well-Suited for Large Biomedical Datasets: Despite the massive size and complexity of MEDLINE, relational database management systems (RDBMS) prove capable of handling indexing and querying efficiently.
- SAX Parser is Crucial for Large XML Datasets: For datasets like MEDLINE, which are distributed as large XML files, the SAX parser's streaming approach is significantly more efficient than DOM, which requires loading the entire document into memory.
- De-normalization for Query Performance: To optimize query speed, the authors adopted a de-normalized schema design, prioritizing read-only performance over strict normalization. This means some data might be repeated across tables to avoid complex joins during queries.
- Trade-offs in Loading and Storage: The choice of how abstracts are stored (as CLOBs vs. truncated text) and the extent of indexing significantly impact disk space utilization. Similarly, parallel vs. sequential processing of input files affects loading times.
- SQL Offers Powerful Querying Beyond PubMed API: A local RDBMS installation unlocks sophisticated querying capabilities not easily achievable through the PubMed API, such as complex aggregations, hierarchical MeSH term analysis, and leveraging full-text search extensions.
Key Results
The paper reports on the performance of three different implementations of their loading tools:
- Loading Times:
- Berkeley (Java, Intel, DB2): 76 hours
- Stanford (Java, Sun, Oracle): 196 hours
- Stanford (Perl, Sun, Oracle): 132 hours
- Disk Space Utilization:
- Berkeley (DB2): 46.3 GB
- Stanford (Oracle): 37.7 GB
- Stanford (Oracle, Perl with flat files): 31.6 GB
These results highlight the variability in performance based on hardware, software, and implementation choices. For instance, the Perl version using intermediate flat files achieved the lowest disk space utilization, partly due to less indexing. The authors also demonstrated sample SQL queries, showing that complex analyses like ranking journals by MeSH term frequency or counting affiliations within a specific timeframe could be executed in minutes or seconds, significantly faster than what might be achievable with the PubMed API for similar tasks.
Limitations and Future Work
Limitations:
- Unpredictable Loading Times: The authors acknowledge that exact loading times are highly dependent on site-specific hardware, software, and network configurations, making precise predictions difficult.
- No Direct PubMed Features: The system does not replicate features like PubMed's "Related Articles" or direct links to full-text articles.
- User Interface: The system is geared towards developers and researchers comfortable with SQL, not end-users who might prefer a more intuitive interface.
Future Work:
- MEDLINE Update Functionality: The authors plan to add functionality to update the database with new MEDLINE versions and update files.
- Alternative Approaches: Continued exploration of alternative approaches for research purposes.
- Open-Source Contribution: The developed database schemas and conversion software are publicly available, encouraging community contributions and improvements. An updated version of the Java code is available, supporting newer MEDLINE DTDs.
Summary updated