Abstract
Feature transformation involves generating a new set of features from the original dataset to enhance the data's utility. In certain domains like material performance screening, dimensionality is large and collecting labels is expensive and lengthy. It highly necessitates transforming feature spaces efficiently and without supervision to enhance data readiness and AI utility. However, existing methods fall short in efficient navigation of a vast space of feature combinations, and are mostly designed for supervised settings. To fill this gap, our unique perspective is to leverage a generator-critic duet-play teaming framework using LLM agents and in-context learning to derive pseudo-supervision from unsupervised data. The framework consists of three interconnected steps: (1) Critic agent diagnoses data to generate actionable advice, (2) Generator agent produces tokenized feature transformations guided by the critic's advice, and (3) Iterative refinement ensures continuous improvement through feedback between agents. The generator-critic framework can be generalized to human-agent collaborative generation, by replacing the critic agent with human experts. Extensive experiments demonstrate that the proposed framework outperforms even supervised baselines in feature transformation efficiency, robustness, and practical applicability across diverse datasets.
Introduction
Feature transformation—the process of creating new features from existing ones (e.g., $f_1 / f_2$)—is a critical step in high-performance machine learning. However, it faces two primary challenges: the astronomical search space of possible feature combinations and the "label bottleneck." Most automated feature engineering tools are supervised, requiring labeled data to evaluate the quality of new features. In fields like drug discovery or material science, labels are often prohibitively expensive or slow to obtain. This research introduces LPFG, a framework designed for Efficient and Unsupervised Feature Transformation (EUFT), navigating massive search spaces without the need for labels.
The LPFG Framework
LPFG utilizes a "duet-play" teaming framework consisting of two specialized Large Language Model (LLM) agents: the Critic Agent and the Generator Agent. This architecture reformulates feature engineering as a generative language task, allowing the system to leverage the pre-trained world knowledge of LLMs.
Agent Roles and Iterative Refinement
- The Critic Agent: This agent performs a dual-layered diagnosis of the dataset. Through Semantic Analysis, it examines feature names to understand their physical or social context. Through Structural Analysis, it evaluates data distributions. The output is a set of "textual gradients"—high-level, actionable advice (e.g., "Try creating a ratio between urea concentration and specific gravity").
- The Generator Agent: Treating features and operators as tokens in a sequence, the Generator produces new feature formulas (e.g.,
(f1 + f2) * f3) guided by the Critic’s advice and in-context examples. - Iterative Refinement: The agents engage in a feedback loop where the Critic evaluates generated features against semantic and structural rules, prompting the Generator to refine its output in subsequent "plays."
This framework also supports human-in-the-loop configurations, where a human expert can replace the Critic agent to steer the transformation process conversationally.
Experimental Evaluation
The framework was evaluated on 12 public datasets from OpenML and Kaggle (including diabetes, airlines, and spaceship). These datasets were selected for their descriptive feature names, which are vital for the Critic agent’s semantic reasoning capabilities.
Key Results
Despite being an unsupervised method, LPFG demonstrated superior performance, outperforming five state-of-the-art supervised baselines, such as AutoFeat and OpenFE, across the majority of test cases.
- Robustness: The generated features improved performance across various downstream models, including Random Forest, XGBoost, and SVM, indicating that the transformations are model-agnostic.
- Efficiency: LPFG maintained stable and short generation times. Unlike traditional search-based methods that scale poorly with dataset size, LPFG’s complexity is primarily tied to LLM inference speed.
- Model Sensitivity: Interestingly, the researchers found that upgrading from GPT-3.5-turbo to GPT-4 provided only marginal gains, suggesting that the framework's duet-play design is the primary driver of its success rather than the raw intelligence of the underlying model.
Conclusion and Future Work
LPFG demonstrates that LLMs can act as domain experts to skip the "blind" search of traditional algorithms. By using textual advice as a proxy for numerical gradients, the framework achieves high-quality, interpretable, and traceable feature transformations in unsupervised settings.
Future work will address the framework's dependency on meaningful metadata (feature names) and explore the potential of human-agent collaborative systems for even more complex industrial datasets. While API costs remain a consideration, the efficiency and performance gains of LPFG represent a significant step forward for unsupervised data readiness.
Summary updated