Symbolic Automated Feature Engineering

33 concepts27 papers1 proposal

In traditional machine learning pipelines, practitioners face a persistent trade-off between predictive performance and interpretability. Deep neural networks and complex ensemble models achieve high accuracy by mapping raw inputs into abstract, high-dimensional latent spaces. However, these latent representations are black boxes, offering no semantic meaning or intuition to domain experts. Conversely, manual feature engineering preserves domain meaning but is a tedious, human-intensive grind that often fails to discover complex, non-linear relationships.

Symbolic Automated Feature Engineering bridges this gap. Instead of projecting data into uninterpretable latent spaces, it automates the discovery of optimal feature representations by restricting the generation process to explicit, human-readable symbolic operations. As argued by Alexandra Zytek et al. in The Need for Interpretable Features: Motivation and Taxonomy, a machine learning model is only as interpretable as its features. By ensuring that automated transformations are symbolic rather than latent, this research direction provides a traceable, "white-box" path to optimization.

Searching the Space of Explicit Mathematical Operations

The core mechanism of symbolic feature engineering relies on treating feature construction as a discrete search problem over a grammar of explicit mathematical operations (such as arithmetic combinations, logarithms, or group-wise aggregations).

One prominent approach uses reinforcement learning (RL) to navigate this transformation space. Udayan Khurana et al. in Feature Engineering for Predictive Modeling Using Reinforcement Learning modeled feature transformations as a graph, utilizing RL agents to learn optimal transformation strategies from past datasets. To make this process collaborative, Fatemeh Nargesian et al. developed Dataset Evolver: An Interactive Feature Engineering Notebook, which allows data scientists to visually inspect and validate these explicit mathematical transformations in real time.

To scale this search to high-dimensional datasets without succumbing to the curse of dimensionality, Meng Xiao et al. introduced cascading actor-critic reinforcement learning agents in Traceable Automatic Feature Transformation via Cascading Actor-Critic Agents and Traceable Group-Wise Self-Optimizing Feature Transformation Learning: A Dual Optimization Perspective. Because these agents select explicit symbolic operations rather than abstract projections, the resulting features remain fully traceable. To address the computational overhead of evaluating these features on downstream models, Tianqi He et al. proposed FastFT: Accelerating Reinforced Feature Transformation via Advanced Exploration Strategies, which uses a novelty-based exploration strategy and a performance predictor proxy to accelerate feature discovery while maintaining a clear mathematical audit trail.

Raw Features ───► [ RL / Evolutionary Search ] ───► Explicit Formulas (e.g., log(A) * B) ───► Interpretable Model

Alternatively, evolutionary computation can be leveraged to "evolve" symbolic expressions. Uday Kamath et al. demonstrated this in Effective Automated Feature Construction and Selection for Classification of Biological Sequences, using genetic programming to discover complex genomic features that capture distant biological interactions while remaining human-readable. As reviewed by Bing Xue et al. in A Survey on Evolutionary Computation for Computer Vision and Image Analysis: Past, Present, and Future Trends, symbolic programming allows evolutionary models to generate explicit mathematical expressions for feature extraction, offering a transparent alternative to deep learning in high-stakes fields like medical imaging.

Grounding Transformations in Domain Logic and Physics

A major advantage of symbolic feature engineering is that the search space can be constrained by domain-specific logic or physical laws. This prevents the automated system from adopting features based on spurious correlations, ensuring that the generated variables serve as actionable scientific hypotheses.

In materials science, Ziyu Xiang 0003 et al. implemented this in Physics-constrained Automatic Feature Engineering for Predictive Modeling in Materials Science by restricting automated feature generation to standard algebraic operations that yield physically valid descriptors. Similarly, Dongjie Wang et al. in Reinforcement Feature Transformation for Polymer Property Performance Prediction applied explicit mathematical operations to raw polymer descriptors. Because the ancestry of each generated feature is physically traceable, researchers can use the discovered formulas to derive new physical laws.

This alignment with domain principles extends beyond the physical sciences:

LLMs as Context-Aware Feature Generators

The latest evolution in this field leverages Large Language Models (LLMs) to bridge the gap between automated code generation and semantic domain knowledge. Instead of searching purely mathematical spaces, these frameworks use LLMs to write feature transformation code accompanied by natural language justifications.

Noah Hollmann et al. pioneered this approach with CAAFE in Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. By utilizing dataset descriptions, CAAFE prompts LLMs to generate Python code for new features based on semantic domain knowledge, ensuring that every new feature is accompanied by a natural language explanation of its intent. To prevent LLMs from introducing ungrounded or spurious features, Mohamed Bouadi et al. developed ReaGen in Synergizing Large Language Models and Knowledge-based Reasoning for Interpretable Feature Engineering, which pairs LLM brainstorming with the logical guardrails of Knowledge Graphs to ensure transformations align with established human expertise.

Dataset Context ───► [ LLM + Knowledge Graph ] ───► Python Code + Text Justification ───► Auditable Feature

Furthermore, Xinhao Zhang et al. demonstrated in Dynamic and Adaptive Feature Generation with LLM that generating natural language justifications for each data transformation creates an auditable trail that addresses the interpretability gap in complex models. This evolutionary optimization capability was formalized by Nikhil Abhyankar et al. in LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers, where human-readable code and natural language reasoning are iteratively refined based on data-driven feedback. To ground these latent features in reality, Hyowon Cho et al. developed the DSAI framework in DSAI: Unbiased and Interpretable Latent Feature Extraction for Data-Centric AI, which establishes a direct link between extracted features and specific data points that exemplify them, creating a verifiable audit trail for model outputs.

Traceability and Semantic Inversion

The ultimate goal of symbolic feature engineering is complete traceability from raw data to downstream prediction. In unstructured domains like legal text, Haoxi Zhong et al. in How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence highlighted how symbolic feature extraction identifies constitutive elements within text to create interpretable intermediate representations. This bridges the gap between raw text and human-understandable logic. Similarly, Shoaib Jameel et al. in Inductive Reasoning about Ontologies Using Conceptual Spaces mapped entities to vector spaces where the dimensions themselves are derived from interpretable features, allowing inductive reasoning systems to ground statistical inferences in symbolic logic.

To ensure that these explanations remain actionable for end-users, Alexandra Zytek et al. proposed the concept of semantic feature inversion in The Need for Interpretable Features: Motivation and Taxonomy. Semantic feature inversion applies mathematical operations that reverse standard preprocessing steps (such as normalization or one-hot encoding), returning the data to its original domain context before presenting explanations to the user.

By combining automated search algorithms with explicit symbolic constraints and semantic inversion, this research direction ensures that machine learning models can achieve high predictive performance without sacrificing transparency.

Key Foundational Work

View Learning for Statistical Relational Learning: With an Application to Mammography (2005) — contributed 1 concept elements to this direction.
Effective Automated Feature Construction and Selection for Classification of Biological Sequences (2014) — contributed 1 concept elements to this direction.
A Machine Learning Approach to Musically Meaningful Homogeneous Style Classification (2014) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 27 papers
  2. Version 1 · 2026-10-04 · 41 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_symbolic_feature_interpretability_8a62a101,
  title = {symbolic feature interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symbolic-feature-interpretability/rdr_8a62a1013a2a42399a9eb7a8b5d6e419},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_symbolic_feature_interpretability_8a62a101_v2,
  title = {symbolic feature interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/symbolic-feature-interpretability/rdr_8a62a1013a2a42399a9eb7a8b5d6e419/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake