Are Large Language Models the New Interface for Data Pipelines?

Sylvio Barbon Junior, Paolo Ceravolo, Sven Groppe, Mustafa Jarrar, Samira Maghool, Florence Sèdes, Soror Sahri, Maurice Van Keulen

TL;DR

This position paper explores the potential of Large Language Models (LLMs) to serve as the primary, natural language interface for complex data pipelines. By examining the intersections between LLMs, Big Data Analytics (BDA), Knowledge Graphs (KGs), Explainable AI (XAI), and Automated Machine Learning (AutoML), the authors outline a vision for more intuitive, human-centric data systems while critically addressing the trade-offs in accuracy and energy consumption.

What Problem Does This Paper Address?

Modern Big Data pipelines are notoriously complex, requiring specialized engineering skills to design, optimize, and maintain. Traditional data modeling approaches often struggle to scale gracefully or remain accessible to non-expert stakeholders.

The authors investigate whether LLMs can democratize access to these pipelines by acting as an intelligent intermediary. Instead of writing complex code or queries, users could interact with data systems using natural language. However, integrating LLMs into production data pipelines introduces significant challenges regarding reliability, factual accuracy, transparency, and computational cost. This paper maps out how adjacent AI fields can cooperatively solve these integration hurdles.

Methods

As a position paper, the authors do not introduce a new empirical algorithm. Instead, they propose a conceptual framework detailing how LLMs can be integrated with four core pillars of modern data ecosystems:

  • Big Data Analytics (BDA): Utilizing LLMs for tasks across the data lifecycle, such as entity resolution, schema matching, and translating natural language queries into robust search queries (e.g., Text-to-SQL).
  • Knowledge Graphs (KGs): Creating a bidirectional synergy where KGs provide factual grounding to reduce LLM hallucinations, and LLMs assist in extracting entities and relations from unstructured text to populate KGs.
  • Explainable AI (XAI): Leveraging LLMs to translate complex model decisions or XAI outputs into easily digestible, natural language summaries for users.
  • Automated Machine Learning (AutoML): Using LLMs to streamline algorithm selection, hyperparameter optimization (HPO), and pipeline synthesis by interpreting user constraints described in plain text.
Figure 1: LLM the interface for data pipelines and their coop- eration to XAI, AutoML and KG.

Datasets

Not discussed in the paper (as it is a conceptual position paper reviewing existing literature).

Key Insights and Ideas

  • The Bidirectional LLM-KG Synergy: KGs act as an anchor for truth. Incorporating them during LLM pre-training or inference can drastically mitigate the "hallucination" problem. Conversely, LLMs excel at the tedious task of parsing text to build and align these very graphs.
  • The "Code Generation" Efficiency Strategy: To combat the massive carbon footprint of LLMs, the authors propose a design convention: use LLMs once to generate static pipeline configurations or efficient code, rather than routing continuous natural language queries through an LLM during runtime execution.
  • AutoML is a Two-Way Street: While LLMs can make AutoML platforms more accessible by explaining hyperparameter impacts in plain English, applying AutoML techniques to LLMs (like Neural Architecture Search for transformers) remains incredibly difficult and cost-prohibitive.

Key Results

While the paper focuses on conceptual synthesis, it highlights several critical benchmarks from recent literature to contextualize the current state of the art:

  • The Text-to-SQL Gap: Even advanced models like GPT-4 only achieve a 54.89% accuracy in executing text-to-SQL tasks, trailing significantly behind the human benchmark of 92.96%.
  • Energy Consumption: A single GPT-4 query is estimated to consume between 0.001 to 0.01 KWh—making it 3.33 to 33.33 times more energy-intensive than a standard Google search.

Limitations and Future Work

The authors openly acknowledge several roadblocks preventing LLMs from fully taking over data pipelines today:

  1. Reliability and Bias: LLMs still suffer from hallucinations and can perpetuate biases present in their training data, making fully automated data pipelines risky without human oversight.
  2. Evaluation Scarcity: Because LLMs are inherently open-ended, definitive ground truths are elusive. The field urgently needs robust, human-centered qualitative evaluation frameworks.
  3. High Computational Costs: The financial and environmental costs of pre-training and running multi-billion parameter models limit their viability for real-time, large-scale data tasks.

Future research directions should focus on domain-specific fine-tuning, prompt engineering to mitigate bias, and developing automated systems that select the most energy-efficient query paths.

Summary updated

Abstract

A Language Model is a term that encompasses various types of models designed to understand and generate human communication. Large Language Models (LLMs) have gained significant attention due to their ability to process text with human-like fluency and coherence, making them valuable for a wide range of data-related tasks fashioned as pipelines. The capabilities of LLMs in natural language understanding and generation, combined with their scalability, versatility, and state-of-the-art performance, enable innovative applications across various AI-related fields, including eXplainable Artificial Intelligence (XAI), Automated Machine Learning (AutoML), and Knowledge Graphs (KG). Furthermore, we believe these models can extract valuable insights and make data-driven decisions at scale, a practice commonly referred to as Big Data Analytics (BDA). In this position paper, we provide some discussions in the direction of unlocking synergies among these technologies, which can lead to more powerful and intelligent AI solutions, driving improvements in data pipelines across a wide range of applications and domains integrating humans, computers, and knowledge.
BibTeX
@inproceedings{junior2024are,
  author = {Sylvio Barbon Junior and Paolo Ceravolo and Sven Groppe and Mustafa Jarrar and Samira Maghool and Florence Sèdes and Soror Sahri and Maurice Van Keulen},
  title = {{Are Large Language Models the New Interface for Data Pipelines?}},
  booktitle = {BiDEDE@SIGMOD},
  year = {2024},
  doi = {10.1145/3663741.3664785},
}
Figure