Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming

Nanxu Gong, Xinyuan Wang, Wangyang Ying, Haoyue Bai, Sixun Dong, Haifeng Chen, Yanjie Fu

Abstract

Feature engineering is a critical yet labor-intensive component of machine learning, often constrained by exponential search spaces and a reliance on labeled data. This paper introduces LPFG (LLM-based Pseudo-Feature Generation), a framework that leverages the internal world knowledge of Large Language Models (LLMs) to perform unsupervised feature transformation. By employing a "duet" of specialized agents—a Critic and a Generator—LPFG diagnoses data distributions and semantic relationships to suggest and execute complex feature transformations. Experimental results across 12 diverse datasets demonstrate that LPFG outperforms traditional supervised methods without requiring any ground-truth labels, offering a scalable and efficient approach to automated feature engineering.

Introduction

High-performance machine learning models frequently depend on feature transformation—the creation of new, informative features from raw data. However, traditional automated feature engineering (AFE) faces two primary challenges: the search space explosion and label scarcity. Most existing tools are supervised, requiring labeled data to evaluate new features, which is often unavailable or expensive to obtain in specialized domains like drug discovery.

LPFG addresses these hurdles by treating feature engineering as an unsupervised task guided by LLMs. By utilizing the semantic reasoning capabilities of LLMs, the framework can infer meaningful interactions between features (e.g., relating "urea concentration" and "specific gravity" in medical datasets) without explicit supervision.

Methodology: The Critic-Generator Duet

The core of LPFG is the duet-play mechanism between two specialized LLM agents. This collaborative framework separates reasoning from execution to enhance performance and focus.

Overview of the LPFG framework showing the interaction between the Critic and Generator agents.

The Critic Agent

The Critic acts as the analytical "brain" of the system. It performs two primary functions:

  1. Semantic Analysis: Interpreting the natural language names of features to understand their real-world context.
  2. Structural Analysis: Evaluating statistical properties such as data distribution and skewness.

Based on these analyses, the Critic generates "textual gradients"—natural language advice that explains why certain transformations (e.g., a log-transform for skewed data) are likely to be beneficial.

The Generator Agent

The Generator serves as the "worker," translating the Critic's advice into actionable symbolic sequences. It treats mathematical operators (e.g., sqrt, log, +, /) and feature IDs as tokens. Guided by the textual gradients, it produces a transformation "recipe" that is subsequently executed via standard scripts.

An example of the Generator agent's output, showing the symbolic feature sequence and the resulting new features for a sample dataset.

Iterative Refinement

The agents engage in a feedback loop where the Critic reviews the Generator's output for logical consistency and alignment with the data's underlying semantics. This iterative process allows the system to refine its suggestions without ever seeing a label.

Experimental Results

The LPFG framework was evaluated on 12 public datasets spanning medical, social, and synthetic domains. Key findings include:

  • Superior Performance: LPFG outperformed five major supervised baselines (including TTG and AutoFeat) on 10 out of 12 datasets. Remarkably, it achieved these gains in an unsupervised setting, whereas the baselines required full access to labels.
  • Efficiency and Scalability: Unlike search-based methods that require training downstream models thousands of times, LPFG's time complexity remains stable. It bypasses the need for repeated model evaluation, making it significantly faster.
  • Model Agnostic: The generated features improved performance across various models, including Random Forest, XGBoost, and SVM, indicating that the transformations capture universal data patterns.

Discussion and Limitations

While LPFG represents a significant step forward in Data-Centric AI, it has limitations. The framework relies heavily on descriptive feature names; anonymous features (e.g., feat_1) neutralize the semantic analysis component. Additionally, extremely large datasets may challenge the context window limits of current LLMs, necessitating a preliminary feature selection step.

Conclusion

LPFG demonstrates that LLMs can serve as effective proxies for human domain experts in feature engineering. By replacing numerical optimization with semantic reasoning and textual gradients, the framework enables high-quality, unsupervised feature transformation that is both efficient and robust.

Summary updated

Abstract

Feature transformation involves generating a new set of features from the original dataset to enhance the data's utility. In certain domains like material performance screening, dimensionality is large and collecting labels is expensive and lengthy. It highly necessitates transforming feature spaces efficiently and without supervision to enhance data readiness and AI utility. However, existing methods fall short in efficient navigation of a vast space of feature combinations, and are mostly designed for supervised settings. To fill this gap, our unique perspective is to leverage a generator-critic duet-play teaming framework using LLM agents and in-context learning to derive pseudo-supervision from unsupervised data. The framework consists of three interconnected steps: (1) Critic agent diagnoses data to generate actionable advice, (2) Generator agent produces tokenized feature transformations guided by the critic's advice, and (3) Iterative refinement ensures continuous improvement through feedback between agents. The generator-critic framework can be generalized to human-agent collaborative generation, by replacing the critic agent with human experts. Extensive experiments demonstrate that the proposed framework outperforms even supervised baselines in feature transformation efficiency, robustness, and practical applicability across diverse datasets.
BibTeX
@inproceedings{gong2025unsupervised,
  author = {Nanxu Gong and Xinyuan Wang and Wangyang Ying and Haoyue Bai and Sixun Dong and Haifeng Chen and Yanjie Fu},
  title = {{Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teaming}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/314},
}
Figure