IMT: A Mixed-Initiative Data Mapping and Search Toolkit

Michael Zang, Adam Gray, Joe Kriege, Kalyan Moy Gupta, David W. AhaAAAI 2008

Introduction

Data interoperability remains a significant challenge for large-scale organizations. When disparate systems must communicate, engineers are tasked with mapping fields between complex database schemas—for instance, linking "Ship_To_Address" in one system to "Delivery_Loc" in another. As schemas expand to include thousands of elements, the number of potential matches grows exponentially ($n \times m$), rendering manual mapping both error-prone and prohibitively expensive.

The Intelligent Mapping Toolkit (IMT) addresses this "n x m" headache through a mixed-initiative approach. Originally developed for the U.S. Transportation Command (USTRANSCOM) to manage a vast web of supply and inventory systems, IMT combines automated AI matching techniques with human expertise to integrate legacy systems significantly faster than traditional manual methods.

System Architecture

The IMT is built upon a distributed multi-agent architecture organized into three distinct layers, facilitating a "mixed-initiative" workflow where the AI suggests potential mappings for human verification.

The three-layer architecture of the Intelligent Mapping Toolkit (IMT).

1. GUI/Application Layer

This layer provides the interface for human-in-the-loop interaction. It presents AI-generated mapping suggestions, allowing experts to provide final semantic verification and acceptance. Beyond mapping, this layer has been refactored to support general search applications.

2. Agent Layer

The core of IMT's intelligence resides in a "federation" of specialized matching agents. Rather than relying on a single algorithm, the system employs multiple strategies simultaneously:

  • N-Gram Matcher: Identifies morphological variations (e.g., "description" vs. "describe").
  • Word Matcher: Performs linguistic analysis on multi-word descriptions.
  • Synonym Matcher: Utilizes domain-specific lexicons to recognize semantic equivalents.
  • Specialized Matchers: Dedicated agents for specific data types, such as geographic locations, measured quantities, and sample data values.

3. Database Layer

This layer serves as the repository for mapping problems, validated solutions, and libraries of abbreviations and synonyms. The system utilizes Case-Based Reasoning (CBR) to learn from previous tasks, improving its similarity assessment performance over time.

Applications and Results

The IMT has been validated through its application to real-world military data environments, specifically the Distribution Process Information Exchange Data Model (DPIEDM)—a massive logical model with thousands of elements.

Key outcomes include:

  • Operational Success: Deployed as a prototype for USTRANSCOM, the system successfully handled large-scale metadata mapping that was previously manually prohibitive.
  • Versatility: A significant finding was that the underlying "Similarity Assessment Service" is general-purpose. By refactoring the GUI into a broader "Application Layer," the technology was reused to create search tools for military reference data and person-of-interest tracking.

Discussion and Future Work

The success of IMT highlights that while fully automated semantic mapping remains unfeasible, a "federated intelligence" approach—using multiple specialized agents—can effectively assist human experts. By handling the heavy lifting of discovery, the system allows experts to focus on high-level semantic decisions.

Current limitations provide a roadmap for future development. While the system identifies how data should map, it does not yet generate the transformation code (e.g., SQL or Java) required for execution. Future efforts are also focused on enhancing instance-based scoring—evaluating matches based on actual data values rather than just metadata names—and establishing more rigorous quantitative benchmarks against other automated schema-matching tools.

Summary updated

BibTeX
@inproceedings{zang2008imt,
  author = {Michael Zang and Adam Gray and Joe Kriege and Kalyan Moy Gupta and David W. Aha},
  title = {{IMT: A Mixed-Initiative Data Mapping and Search Toolkit}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2008},
}
Figure