TL;DR
This paper introduces FSTO-Gen, a novel framework that leverages Large Language Models (LLMs) to automatically generate a hierarchical ontology of "Functional Semantic Types" (FSTs) for tabular data. This ontology not only annotates data with semantic meaning but also generates executable Python code for normalization, validation, and casting, significantly streamlining the data onboarding process.
What Problem Does This Paper Address?
Onboarding tabular data into data science workflows is notoriously time-consuming and error-prone. This process involves understanding the meaning of columns, their formats, units, and how they relate to other data. Humans spend significant effort normalizing data, handling inconsistent formats, implicit units (like currency), and recasting values for joins or comparisons. Existing methods often rely on pre-defined ontologies or struggle to generate the actual code needed for these tasks. The authors aim to automate this entire process, from semantic annotation to functional code generation, by harnessing the power of LLMs.
Methods
The core of FSTO-Gen is a multi-stage process that uses LLMs to build a hierarchical ontology:
- Table to T-FST (Table-level Functional Semantic Type): For each table, an LLM analyzes column names, descriptions, and data statistics (mean, std, quantiles, etc.) to identify semantic entities. It then generates Python class definitions for T-FSTs, including descriptive attributes and a
cast()method to transform column values into a canonical format. - T-FST to P-FST (Product-level Functional Semantic Type): Within a "product" (a collection of related tables), identically named T-FSTs are agglomerated into a single P-FST. The system selects the T-FST that best handles the data (minimizing errors and maximizing complex transformations) as the representative P-FST.
- P-FST to G-FST (General Functional Semantic Type): P-FSTs across different products with similar names are further aggregated into G-FSTs. The LLM generates a
super_cast()method that can handle the outputs from various P-FSTcast()methods and avalidate()method for sanity checks. - G-FST to G-FST (Cross-type Casting): To enable data fusion and discovery, the system identifies semantically similar G-FSTs (even if named differently) and generates
cross_type_cast()functions. This allows for non-trivial joins and transformations between different semantic types. LLMs are used to determine which G-FSTs are convertible and to generate the corresponding Python code.
Datasets
The authors evaluated FSTO-Gen on three data universes:
- Kaggle: Sourced from the top 1000 most downloaded Kaggle datasets, selecting 707 datasets.
- Harvard Dataverse: Selected 484 most downloaded datasets, requiring additional preparation due to null columns.
- FData: A proprietary commercial financial dataset with 428 tables.
These datasets were curated to include tables with at least 80% of columns having more than one non-null value. The paper does not introduce new benchmarks but uses existing data sources.
Key Insights and Ideas
- LLMs as Code Generators for Data Semantics: The paper demonstrates that LLMs can go beyond simple text generation to produce functional Python code that embodies semantic understanding of tabular data. This is a significant step towards automating complex data wrangling tasks.
- Hierarchical Ontology for Generalization: The multi-layered ontology (T-FST, P-FST, G-FST) allows for progressive generalization, moving from table-specific types to universally applicable semantic types. This structure is key to discovering relationships across diverse datasets.
- Functional Semantic Types: The core innovation is the concept of FSTs, which are not just static type labels but active components that include executable code for data manipulation. This bridges the gap between semantic understanding and practical application.
- Automated Cross-Type Casting: The ability to automatically generate
cross_type_cast()functions is crucial for enabling data fusion and discovery, allowing disparate datasets to be meaningfully joined and analyzed. - LLM Hallucinations and Conservatism: The authors highlight that LLMs can sometimes hallucinate or make overly broad assumptions (e.g., assuming two types are castable when they are not). They emphasize the need for conservative conversions and suggest strategies like consensus-building to mitigate these issues.
Key Results
- High Precision and Recall in P-FST Generation: Human evaluation showed that LLMs achieved high precision and recall in recognizing semantic entities and generating correctly scoped and functionally correct P-FSTs.
- Effective Normalization and Validation: The generated FSTs successfully performed normalization and validation tasks, with a high percentage of non-null values passing through unchanged or undergoing complex transformations.
- Cross-Type Casting Success: The system demonstrated success in generating
cross_type_cast()functions, enabling transformations between semantically similar G-FSTs. - Cost Efficiency: The LLM-dependent parts of FSTO-Gen were analyzed, showing a manageable number of LLM calls and minimal human correction required for code compilation.
Limitations and Future Work
- Domain-Specific Datasets: FSTO-Gen is less successful with highly domain-specific datasets where LLMs may lack familiarity with specialized terminology, leading to incorrectly scoped FSTs.
- LLM Hallucinations: The tendency for LLMs to hallucinate or make incorrect assertions (e.g., in
cross_type_cast()logic) is a significant limitation. - Future Work: The authors suggest exploring fine-tuning LLMs and retrieval-augmented generation to improve performance on domain-specific data. They also propose using consensus mechanisms (e.g., batching outputs and performing a consensus) to mitigate hallucination issues. Further research into using validation feedback to improve normalization is also suggested.
Summary updated