GDC-SM: The GDC Schema Matching Benchmark

Yurong Liu, Eduardo H. M. Pena, Aécio Santos, Ex Wu, Juliana FreirearXiv 2024 · DOI · arXiv

This blog post summarizes the paper "Magneto: Combining Small and Large Language Models for Schema Matching" by Liu et al., set to appear at PVLDB 2025.

TL;DR

Magneto is a hybrid framework for schema matching that combines the efficiency of Small Language Models (SLMs) with the reasoning power of Large Language Models (LLMs). By using LLMs to generate synthetic training data for SLMs and then using the LLM as a precise reranker, Magneto achieves state-of-the-art accuracy on complex biomedical datasets without requiring manually labeled training data.

What Problem Does This Paper Address?

Schema matching (SM)—the process of identifying correspondences between columns in different tables—is a fundamental bottleneck in data integration. Imagine trying to merge ten different clinical studies: one table calls a column patient_age, another uses age_at_diagnosis, and a third uses days_to_birth.

Traditional methods rely on string similarity or basic statistics, which fail when names are cryptic or semantically ambiguous. Modern approaches using Language Models (LMs) face a "Goldilocks" problem:

  1. Small LMs (SLMs): Efficient but require massive amounts of manually curated, domain-specific training data to perform well.
  2. Large LMs (LLMs): Highly capable but prohibitively expensive and slow for large-scale schemas. They also struggle with "context window" limits when a target schema has hundreds of columns.

Magneto aims to find the "just right" balance: high accuracy, low cost, and the ability to generalize to complex domains like biomedicine.

Methods

Magneto introduces a two-phase pipeline: Candidate Retrieval and Reranking.

  • Phase 1: Candidate Retrieval (SLM): Magneto uses an SLM (like MPNet) to convert column names and sampled values into vector embeddings. It quickly identifies the top-$k$ most likely matches from the target schema.
  • Phase 2: Reranking (LLM): The top-$k$ candidates are sent to an LLM (like GPT-4o-mini). The LLM performs a "deep dive," using a specialized Scoring-Oriented Prompt to assign a similarity score (0.0 to 1.0) to each pair. This allows the model to catch subtle semantic nuances the SLM might miss.

The "Secret Sauce": LLM-Powered Fine-Tuning To solve the lack of training data for SLMs, the authors use an LLM to generate it. The LLM takes a target column and creates syntactically diverse but semantically equivalent "synthetic" versions (e.g., generating tumor_site from tissue_source_sites). The SLM is then fine-tuned using contrastive learning (triplet loss) on this synthetic data, teaching it to group related concepts together in vector space without a single human label.

Datasets

The authors evaluated Magneto on two main fronts:

  1. GDC-SM (New Benchmark): A challenging new benchmark created with biomedical experts. It involves mapping real-world cancer research data to the National Cancer Institute’s Genomics Data Commons (GDC) standard, which features over 700 target attributes.
  2. Valentine Benchmark: A standard industry benchmark including datasets like Magellan (e-commerce), WikiData, and ChEMBL (chemical biology).

Key Insights and Ideas

  • Don't Use a Sledgehammer for Every Nail: You don't need an LLM to compare every single column pair. Using an SLM as a filter reduces the LLM's workload by orders of magnitude, making the process scalable to schemas with hundreds of columns.
  • LLMs as Teachers: One of the most powerful ideas here is using the LLM's vast general knowledge to "teach" the smaller model through synthetic data generation. This bypasses the need for expensive manual annotation.
  • Contextual Anchors: The paper highlights that column headers alone are often insufficient. By using Priority Sampling, Magneto includes the most representative data values in the prompt, providing the "anchors" necessary for the model to distinguish between similar-looking columns.

Key Results

  • Superior Accuracy: On the difficult GDC benchmark, Magneto (specifically the ft-llm variant) outperformed traditional tools like COMA and supervised models like Unicorn, achieving significantly higher Mean Reciprocal Rank (MRR).
  • Scalability: While other state-of-the-art models (like ISResMat) failed to complete the GDC task after 100 columns, Magneto maintained stable runtimes, finishing large tasks in minutes.
  • Zero-Shot Strength: Even without fine-tuning, Magneto’s reranking strategy outperformed many existing specialized models, proving the robustness of their prompting and serialization techniques.

Limitations and Future Work

  • Retrieval Bottleneck: Magneto’s reranker is only as good as the candidates it receives. If the correct match isn't in the SLM's top-$k$ list, the LLM never sees it. The authors suggest exploring hybrid retrieval strategies to mitigate this.
  • Prompt Sensitivity: Like all LLM-based methods, performance can vary based on prompt structure. Future work may involve using frameworks like DSPy to automatically optimize these prompts.
  • Domain Expansion: While highly effective in biomedicine, the authors plan to test the framework on even broader, multi-domain datasets to further improve the SLM's "out-of-the-box" retrieval capabilities.

Summary updated

Abstract

GDC-SM is a schema matching evaluation benchmark based on a real data harmonization scenario that is common in biomedical research: pooling datasets from multiple studies to increase the number of patients. We used the study by Li et al. (2023) in which the authors: 1) pooled data from ten studies (available as supplementary material in the original primary research papers) that cover multiple patient cohorts and cancer types; and 2) mapped each dataset into the GDC standard, a data standard set by the US National Cancer Institute's Genomics Data Commons (GDC) for cancer genomic data. We obtained datasets from the 10 studies related to tumor analysis (see table below), and, with the help of the biomedical experts, manually aligned and matched these datasets to the Genomics Data Commons (GDC) standard version 3.1.0. The GDC standard dictionary describes data using a graph model that includes names and descriptions for nodes and attributes, as well as acceptable values for some attributes. However, for compatibility with other schema matching methods and benchmarks, we transformed the model to a relational schema that contains only column names and domain information (i.e., we disregard column descriptions). We created a simplified table reflecting the GDC format, the "target" table, listing domain values for each column without repetition. The benchmark includes 10 pairs of input-target tables. The number of columns in the source tables ranges from 16 to 179, and the number of rows ranges from 93 to 225. Our simplified GDC target schema comprises a single table with 736 columns. While some columns have a small number of distinct values (e.g., binary yes/no attributes), some contain up to 4,478 distinct values. The ground truth was manually curated by multiple annotators -- at least 3 annotators per matching pair, who used a mix of manual and automated methods for identifying possible candidate matches (e.g., GDC search tools and bdi-kit). Given that the correctness of some matches is very challenging to determine even for bioinformatics experts (e.g., it may require reading the original papers or asking data producers), the final match decisions were made by consensus based on what users would expect from an algorithm given the limited context. For more details, refer to our paper: Yurong Liu, Eduardo Pena, Aécio Santos, Eden Wu, and Juliana Freire. "Magneto: Combining Small and Large Language Models for Schema Matching." To appear at PVLDB 2025. Arxiv preprint 2412.08194. For citations, please consider citing the paper above and this dataset as described below: Santos, A., Wu, E., Lopez, R., Keegan, S., Pena, E., Liu, W., Liu, Y., Fenyo, D., & Freire, J. (2025). GDC-SM: The GDC Schema Matching Benchmark (1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.14963588 How to use The raw datasets used in this benchmark are sourced from publicly-available studies under Creative Commons licenses. However, some of these licenses include the NoDerivatives (ND) restriction, which prevents others from redistributing modified versions of the material, including transformed or processed datasets. Therefore, we only provide the ground-truth annotations and information about how to reconstruct the data tables from their original data source. We describe below how to recreate the CSV table files. Generating data files from the publisher's original data For convenience, we provide the gdc_download.py script, which automates the process of downloading, extracting, and preparing the data. It includes the following steps: Download Data: It retrieves the dataset from the paper publisher's website (supplemental files distributed with the paper). Extract Data: If the file is in Excel format, it extracts the specified sheet to a dataframe file. Clean Data: Performs some cleaning steps to ensure datasets are the same as the ones used in the paper (e.g., for the Vasaikar.csv dataset, it removes the first and last rows). Save as CSV: The processed data is saved as a .csv file in the input-tables folder. Install the dependencies listed in the requirements.txt file and run the script, and it will automatically handle the downloading, extraction, and processing of the data for you. Included files The dataset release includes the following files: gdc-sm-data.zipContains manually curated ground-truth mappings and a standardized target schema based on a real-world biomedical data harmonization task. It includes: data/ground-truth/Ten CSV files (Cao.csv, Clark.csv, Dou.csv, Gillette.csv, Huang.csv, Krug.csv, McDermott.csv, Satpathy.csv, Vasaikar.csv, Wang.csv), each containing mappings between column names from a source dataset and attributes in the GDC target schema. These mappings serve as the manually curated ground truth for the schema matching task. data/target-tables/gdc_unique_columns_concat_values.csv, the GDC target schema formatted as a relational table. It lists all attributes from the GDC data dictionary along with their unique, concatenated domain values. gdc_download.pyA Python script that automates the download, extraction, and preparation of the source datasets directly from the original publishers. It handles Excel-to-CSV conversions, minor data cleaning steps, and file organization. When executed, this script creates two additional folders as output: data/downloads/Contains the original spreadsheet files downloaded from the paper publisher's website. data/input-tables/Contains tables, in CSV format, to be used as source tables for the schema matching task. papers_info.jsonA metadata file listing key details for each source study, including publication titles, URLs to the papers, dataset links, and sheet names from which the benchmark datasets were extracted. Needed needed to run the gdc_download.py. requirements.txtSpecifies the Python package dependencies needed to run gdc_download.py . Source table information The table below includes details for each study, such as relevant URLs for the original spreadsheet files used in the benchmark and sheet names where the source tables appear. Each row corresponds to a study and provides the following information: First Author: Name of the study's first author. Paper Title: Title of the publication. Paper URL: Link to the paper. Dataset URL: Link to the supplementary file containing the dataset extract from the paper for convenience. Relevant File: Name of the file or table in the supplementary material. Sheet Name: Name of the sheet in the file that contains the data. First Author Paper Title Paper URL Dataset URL Sheet Name Clark, D.J. Integrated Proteogenomic Characterization of Clear Cell Renal Cell Carcinoma https://doi.org/10.1016/j.cell.2019.10.007 https://ars.els-cdn.com/content/image/1-s2.0-S0092867419311237-mmc1.xlsx ccrcc_clinical_characteristics Krug, K. Proteogenomic Landscape of Breast Cancer Tumorigenesis and Targeted Therapy https://doi.org/10.1016/j.cell.2020.10.036 https://ars.els-cdn.com/content/image/1-s2.0-S0092867420314008-mmc1.xlsx A) Metadata Vasaikar, S. Proteogenomic Analysis of Human Colon Cancer Reveals New Therapeutic Opportunities https://doi.org/10.1016/j.cell.2019.03.030 https://ars.els-cdn.com/content/image/1-s2.0-S0092867419302922-mmc1.xlsx B-ClinicalData Wang, L.-B. Proteogenomic and metabolomic characterization of human glioblastoma https://doi.org/10.1016/j.ccell.2021.01.006 https://ars.els-cdn.com/content/image/1-s2.0-S1535610821000507-mmc2.xlsx clinical_data Huang, C. Proteogenomic insights into the biology and treatment of HPV-negative head and neck squamous cell carcinoma https://doi.org/10.1016/j.ccell.2020.12.007 https://ars.els-cdn.com/content/image/1-s2.0-S1535610820306553-mmc2.xlsx Meta table Satpathy, S. A proteogenomic portrait of lung squamous cell carcinoma https://doi.org/10.1016/j.cell.2021.07.016 https://ars.els-cdn.com/content/image/1-s2.0-S0092867421008576-mmc1.xlsx Table S1A Gillette, M.A. Proteogenomic Characterization Reveals Therapeutic Vulnerabilities in Lung Adenocarcinoma https://doi.org/10.1016/j.cell.2020.06.013 https://ars.els-cdn.com/content/image/1-s2.0-S0092867420307443-mmc1.xlsx Annotions_S1A McDermott, J.E. Proteogenomic Characterization of Ovarian HGSC Implicates Mitotic Kinases, Replication Stress in Observed Chromosomal Instability https://doi.org/10.1016/j.xcrm.2020.100004 https://ars.els-cdn.com/content/image/1-s2.0-S2666379120300045-mmc2.xlsx BaselineClinicalData Cao, L. Proteogenomic characterization of pancreatic ductal adenocarcinoma https://doi.org/10.1016/j.cell.2021.08.023 https://ars.els-cdn.com/content/image/1-s2.0-S0092867421009971-mmc1.xlsx Clinical_data Dou, Y. Proteogenomic Characterization of Endometrial Carcinoma https://doi.org/10.1016/j.cell.2020.01.026 https://ars.els-cdn.com/content/image/1-s2.0-S0092867420301070-mmc1.xlsx UCEC_CPTAC3_meta_table_V2.1 Acknowledgements This work was supported by NSF awards IIS-2106888 and OAC-2411221, and the DARPA Automating Scientific Knowledge Extraction and Modeling (ASKEM) program, agreement No. HR0011262087, and the ARPA-H BDF program. The views, opinions, and findings expressed are those of the authors and should not be interpreted as representing the official views or policies of DARPA, ARPA-H, the U.S. Government, or NSF. We thank our collaborators, Dr. Tonia Korves and the MITRE team, for contributing their experiences in biomedical data integration and an initial proof of concept of this benchmark.
BibTeX
@misc{liu2024gdc,
  author = {Yurong Liu and Eduardo H. M. Pena and Aécio Santos and EX Wu and Juliana Freire},
  title = {{GDC-SM: The GDC Schema Matching Benchmark}},
  year = {2024},
  eprint = {2412.08194},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.08194},
}
Figure