GUDM: Automatic Generation of Unified Datasets for Learning and Reasoning in Healthcare

Rahman Ali, Muhammad Hameed Siddiqi, Muhammad Idris, Taqdir Ali, Shujaat Hussain, Eui-Nam Huh, Byeong Ho Kang, Sungyoung LeeSensors 2015 · DOI

TL;DR

This paper introduces GUDM (Global Unified Data Model) and a "data modeler" tool to automatically integrate diverse healthcare datasets into a single, unified format. This is crucial for enabling machine learning and reasoning applications in healthcare, significantly reducing the time and effort required for data preparation.

What Problem Does This Paper Address?

Healthcare generates vast amounts of data from various sources like clinical trials, patient records, social media, and sensor devices. However, this data is often heterogeneous, with different formats, naming conventions, and semantic meanings. Integrating these disparate sources into a single, usable dataset for analysis and decision support is a significant challenge. Existing tools often require manual effort and expertise, making the process time-consuming and prone to errors. This paper aims to automate this integration process, specifically addressing the challenges of semantic heterogeneity, overlapping attributes, and data fusion.

Methods

The core of the proposed solution is the Global Unified Data Model (GUDM) and an associated "data modeler" tool. The process involves several key steps:

  • Importing Heterogeneous Data Sources: The tool supports various input formats (.csv, .xls, .tsv, .accdb) and allows knowledge engineers to import multiple datasets.
  • Preprocessing: This phase includes:
  • Filling Missing Values: Uses a common technique to replace missing values with the most common value (nominal) or mean (numeric).
  • Finding Intrinsic Relationships: Sorts datasets based on IDs to ensure one-to-one relationships between instances across datasets.
  • Reduction: Ensures all datasets have an equal number of records by reducing them to the size of the smallest dataset (baseline dataset).
  • Prioritization Phase: An expert-centric priority-based approach is employed. Domain experts assign priorities to datasets based on their credibility. This is crucial for resolving conflicts when attributes overlap.
  • Defining a Global Unified Data Model: The tool automatically creates a unified data model by sequentially adding attributes from all participating datasets, ensuring that overlapping attributes are added only once.
  • Logging Data to the Unified Data Model:
    • Unique Attributes: Data is directly transferred.
  • Overlapping Attributes: The priority-based approach is used. If datasets have different priorities, the data from the higher-priority dataset is selected. If priorities are equal, a "completeness" measure (fewer missing values) is used to break ties.
  • Exporting Unified Dataset: The unified dataset can be exported in various formats compatible with common data analysis tools (e.g., WEKA, KEEL, ROSE2), with appropriate data types assigned to attributes.

Datasets

The paper uses a simulated case study involving diabetes mellitus data. The datasets used are:

  • Clinical Dataset: Acquired from a local hospital, containing 100 patient records with attributes like ID, Gender, Total Cholesterol (TC), Triglyceride (TG), LDL, HDL, SGOT, SGPT, FBS, SBP, DBP, Weight, Height, Hypoglycemia, HbA1c, and Diabetes type.
  • Social Media Dataset: Obtained from a Social Media and Interaction Engine (SMIE), containing 50 records with attributes like ID, Gender, Symptoms, Sentiments, and Activities.
  • Sensory Dataset: Acquired using embedded smartphone sensors, containing physical activity data with attributes like ID and Activities.

The evaluation also involved six sets of locally generated diverse datasets with varying complexity levels, used to assess the "data modeler" tool.

Key Insights and Ideas

  • Automation is Key: The primary insight is that automating the complex and time-consuming process of data integration is essential for advancing healthcare analytics.
  • Expert-Centric Prioritization: Leveraging domain expert knowledge to prioritize datasets is a practical and effective way to handle data heterogeneity and attribute overlaps.
  • Completeness as a Tie-Breaker: Using dataset completeness (fewer missing values) as a secondary criterion for prioritizing datasets when priorities are equal is an intelligent way to ensure data quality.
  • Early Integration: The approach emphasizes early integration, resolving inconsistencies before combining datasets, which is more robust than late-stage fusion.
  • Domain Independence: While demonstrated for healthcare, the GUDM and "data modeler" tool are designed to be domain-independent, offering broader applicability.

Key Results

The evaluation demonstrated significant time savings. The "data modeler" tool reduced the average time for unified dataset creation by 84.1% compared to traditional manual methods (using MS Excel). Specifically, it saved 81.9% of expert time and 84.9% of knowledge engineer time. The tool successfully integrated three diverse datasets (clinical, social media, sensory) for a diabetes scenario, resulting in a unified dataset with 19 attributes.

Limitations and Future Work

  • Limitations:
    • Dynamic Priority Assignment: The current priority assignment is static and relies on user input.
  • Attribute-Level Priorities: The approach focuses on dataset-level priorities, not attribute-level.
  • Synonymous/Polysemous Attributes: The system does not automatically resolve attributes with different names but the same meaning (synonyms) or the same name but different meanings (polysemous).
  • One-to-Many Relationships: Handling one-to-many relationships between datasets is left for future work.
  • In-Memory Implementation: The current implementation is in-memory, which might limit its scalability for very large datasets.
  • Tab-Separated Input: Support for tab-separated text files as input is limited.
  • Future Work:
    • Develop methods for integrating more diverse data sources, including imaging data.
  • Create a dynamic strategy to automatically measure dataset and attribute importance.
  • Design a more advanced editor to handle multiple file formats and extend system usability.
  • Address the resolution of polysemous and synonymous attributes.
  • Resolve conflicts between columns of different datasets and enable fusion for datasets with one-to-many relationships.
  • Explore solutions for handling large datasets that exceed in-memory capacity.

Summary updated

BibTeX
@article{ali2015gudm,
  author = {Rahman Ali and Muhammad Hameed Siddiqi and Muhammad Idris and Taqdir Ali and Shujaat Hussain and Eui-Nam Huh and Byeong Ho Kang and Sungyoung Lee},
  title = {{GUDM: Automatic Generation of Unified Datasets for Learning and Reasoning in Healthcare}},
  journal = {Sensors},
  year = {2015},
  doi = {10.3390/s150715772},
}
Figure