Termite: a system for tunneling through heterogeneous data

Raul Castro Fernandez, Samuel Madden

Overview

Researchers from MIT’s CSAIL have developed Termite, a database system designed to overcome the persistent hurdles of data integration. Rather than forcing diverse data sources into a rigid, unified schema, Termite maps both structured and unstructured data into a shared vector space. This approach enables users to discover related information through proximity-based "joins," effectively bypassing the labor-intensive manual cleaning typically required for heterogeneous datasets.

The Data Integration Challenge

For decades, the primary goal of data management has been seamless integration: the ability to query across disparate databases and file formats as a single entity. Traditionally, this requires a uniform schema, exact value matching, and extensive data cleaning—for instance, ensuring that "MIT" and "Massachusetts Institute of Technology" are recognized as identical.

This process is notoriously expensive and time-consuming, leaving vast quantities of organizational data "locked" in silos. Termite proposes a "dirt-loving" philosophy: instead of striving for perfect schemas, the system embraces "dirty" data by finding a common mathematical representation that captures the underlying relatedness of information, regardless of its original format.

The Termite Pipeline

Termite processes data through a sophisticated five-stage pipeline designed to transform raw inputs into a searchable embedding space.

The five-stage Termite pipeline for data extraction, encoding, and embedding.
  1. Extraction: Connectors convert raw data (SQL tables, PDFs, JSON) into "triples" (subject-predicate-object). A database row might be represented as John:value - age:attribute - 22:value.
  2. Encoding: To manage massive vocabularies, Termite employs hashing-based dictionary encoding, mapping terms to fixed-size vectors to enhance performance over traditional one-hot encoding.
  3. Learning (Deep Metric Learning): At the system's core, a Siamese Network learns embeddings by training on pairs of entities. "Positive" pairs (entities appearing together in a triple) are moved closer in the vector space, while "negative" pairs are pushed apart.
  4. Refinement: To combat "hubness"—the tendency for certain points to appear close to many others—Termite calculates a hubness factor to filter out noisy results during queries.
  5. Serving: The system introduces the Termite-Join operator. Unlike standard SQL joins requiring exact equality, Termite-Join retrieves the $K$ closest entities in the embedding space based on vector distance.

Experimental Results

The system was evaluated using a custom dataset focused on CSAIL Faculty, incorporating structured data from the MIT Data Warehouse and DBPedia alongside unstructured data from Wikipedia and faculty webpages. Benchmarks included record linkage and concept expansion tasks for 52 faculty members.

Performance comparison between Full-Text Search (FTS) and Termite (TMT) across two tasks.

In a head-to-head user study against a standard Full-Text Search (FTS) engine (Elasticsearch), Termite demonstrated superior effectiveness. For tasks involving faculty contributions, Termite users achieved 100% recall in just 4.2 minutes, whereas FTS users only captured 27% of results in 6.1 minutes. Additionally, the system successfully identified 77% of varying representations for the same individual (e.g., linking "David DeWitt" to "Dave DeWitt") without manual configuration.

Key Insights and Future Directions

The power of Termite lies in shifting from binary joins to continuous, proximity-based joins. This allows the system to relate a database record to a mention in a news article even without explicit links. By focusing on embeddings, the system becomes robust to synonyms and varying schemas, learning that terms like "Turing Award" and "Godel Prize" are related through their shared contexts.

While promising, the system faces challenges such as storage overhead—mapping tokens to high-dimensional vectors can increase storage needs by up to 20x. Future work will explore dimensionality reduction, faster training architectures like Skip-gram, and the use of Topological Data Analysis (TDA) to better understand the geometric "shape" of data in vector space.

Summary updated

Abstract

Data-driven analysis is important in virtually every modern organization. Yet, most data is underutilized because it remains locked in silos inside of organizations; large organizations have thousands of databases, and billions of files that are not integrated together in a single, queryable repository. Despite 40+ years of continuous effort by the database community, data integration still remains an open challenge.In this paper, we advocate a different approach: rather than trying to infer a common schema, we aim to find another common representation for diverse, heterogeneous data. Specifically, we argue for an embedding (i.e., a vector space) in which all entities, rows, columns, and paragraphs are represented as points. In the embedding, the distance between points indicates their degree of relatedness. We present Termite, a prototype we have built to learn the best embedding from the data. Because the best representation is learned, this allows Termite to avoid much of the human effort associated with traditional data integration tasks. On top of Termite, we have implemented a Termite-Join operator, which allows people to identify related concepts, even when these are stored in databases with different schemas and in unstructured data such as text files, webpages, etc. Finally, we show preliminary evaluation results of our prototype via a user study, and describe a list of future directions we have identified.
BibTeX
@inproceedings{fernandez2019termite,
  author = {Raul Castro Fernandez and Samuel Madden},
  title = {{Termite: a system for tunneling through heterogeneous data}},
  booktitle = {aiDM@SIGMOD},
  year = {2019},
  doi = {10.1145/3329859.3329877},
}
Figure