Abstract
Feature engineering is a critical yet labor-intensive component of machine learning, often constrained by exponential search spaces and a reliance on labeled data. This paper introduces LPFG (LLM-based Pseudo-Feature Generation), a framework that leverages the internal world knowledge of Large Language Models (LLMs) to perform unsupervised feature transformation. By employing a "duet" of specialized agents—a Critic and a Generator—LPFG diagnoses data distributions and semantic relationships to suggest and execute complex feature transformations. Experimental results across 12 diverse datasets demonstrate that LPFG outperforms traditional supervised methods without requiring any ground-truth labels, offering a scalable and efficient approach to automated feature engineering.
Introduction
High-performance machine learning models frequently depend on feature transformation—the creation of new, informative features from raw data. However, traditional automated feature engineering (AFE) faces two primary challenges: the search space explosion and label scarcity. Most existing tools are supervised, requiring labeled data to evaluate new features, which is often unavailable or expensive to obtain in specialized domains like drug discovery.
LPFG addresses these hurdles by treating feature engineering as an unsupervised task guided by LLMs. By utilizing the semantic reasoning capabilities of LLMs, the framework can infer meaningful interactions between features (e.g., relating "urea concentration" and "specific gravity" in medical datasets) without explicit supervision.
Methodology: The Critic-Generator Duet
The core of LPFG is the duet-play mechanism between two specialized LLM agents. This collaborative framework separates reasoning from execution to enhance performance and focus.
The Critic Agent
The Critic acts as the analytical "brain" of the system. It performs two primary functions:
- Semantic Analysis: Interpreting the natural language names of features to understand their real-world context.
- Structural Analysis: Evaluating statistical properties such as data distribution and skewness.
Based on these analyses, the Critic generates "textual gradients"—natural language advice that explains why certain transformations (e.g., a log-transform for skewed data) are likely to be beneficial.
The Generator Agent
The Generator serves as the "worker," translating the Critic's advice into actionable symbolic sequences. It treats mathematical operators (e.g., sqrt, log, +, /) and feature IDs as tokens. Guided by the textual gradients, it produces a transformation "recipe" that is subsequently executed via standard scripts.
Iterative Refinement
The agents engage in a feedback loop where the Critic reviews the Generator's output for logical consistency and alignment with the data's underlying semantics. This iterative process allows the system to refine its suggestions without ever seeing a label.
Experimental Results
The LPFG framework was evaluated on 12 public datasets spanning medical, social, and synthetic domains. Key findings include:
- Superior Performance: LPFG outperformed five major supervised baselines (including TTG and AutoFeat) on 10 out of 12 datasets. Remarkably, it achieved these gains in an unsupervised setting, whereas the baselines required full access to labels.
- Efficiency and Scalability: Unlike search-based methods that require training downstream models thousands of times, LPFG's time complexity remains stable. It bypasses the need for repeated model evaluation, making it significantly faster.
- Model Agnostic: The generated features improved performance across various models, including Random Forest, XGBoost, and SVM, indicating that the transformations capture universal data patterns.
Discussion and Limitations
While LPFG represents a significant step forward in Data-Centric AI, it has limitations. The framework relies heavily on descriptive feature names; anonymous features (e.g., feat_1) neutralize the semantic analysis component. Additionally, extremely large datasets may challenge the context window limits of current LLMs, necessitating a preliminary feature selection step.
Conclusion
LPFG demonstrates that LLMs can serve as effective proxies for human domain experts in feature engineering. By replacing numerical optimization with semantic reasoning and textual gradients, the framework enables high-quality, unsupervised feature transformation that is both efficient and robust.
Summary updated