Interactive Reinforcement Learning for Symbolic Regression from Multi-Format Human-Preference Feedbacks

Laure Crochepierre, Lydia Boudjeloud-Assala, Vincent BarbesantIJCAI 2022 · DOI

TL;DR

This paper introduces an interactive platform for symbolic regression that leverages human preferences to guide the search for mathematical expressions. By allowing users to provide feedback in various intuitive ways, the system learns to generate more relevant and interpretable symbolic models, especially for complex real-world problems where the underlying formula is unknown.

What Problem Does This Paper Address?

Symbolic Regression (SR) aims to discover a mathematical expression that accurately represents the relationship between input variables and a target output. Traditionally, SR relied on methods like Genetic Programming. While Deep Learning and Reinforcement Learning (RL) have shown promise, they often struggle with interpretability and require vast amounts of data. Furthermore, in many real-world applications, the exact underlying formula is unknown, making SR an exploratory tool. The challenge is to make this exploration more efficient and guided by human expertise, especially when the user might have domain knowledge but not the precise mathematical form. Existing interactive methods can be tedious or require specialized knowledge, making them less accessible.

Methods

The core of this work is an interactive Reinforcement Learning approach called "interactive RBG2-SR" (Reinforcement Based Grammar Guided Symbolic Regression). The system uses a grammar to constrain the generation of symbolic expressions, ensuring they adhere to predefined rules.

Here's a breakdown of the key components:

  • Reinforcement Learning (RL) Agent: A neural agent learns to generate symbolic expressions. It's trained using the REINFORCE algorithm with a risk-seeking policy.
  • Grammar-Guided Generation: Symbolic expressions are built as trajectories constrained by grammatical rules (defined in Backus-Naur Form - BNF). This ensures the generated expressions are syntactically valid.
  • Human Preference Feedback: This is the novel aspect. Instead of relying solely on a quantitative reward signal (like Mean Squared Error), the system actively solicits human feedback. Users can provide preferences in three ways:
  • Categorization: Labeling expressions as "Best," "Average," or "Bad." This is an efficient way to gather broad preferences.
  • Pairwise Comparison: Directly choosing between two expressions.
  • Suggestion/Improvement: Users can suggest improvements to an expression by iteratively building a new one using the grammar.
  • Preference-Based Learning: The RL agent's objective function is modified to maximize the probability of generating preferred trajectories and avoid non-preferred ones, using a weighted pairwise disagreement loss.

Datasets

The paper mentions using:

  • State-of-the-art benchmarks: Specifically, the Nguyen SR benchmark is mentioned for testing the system's ability to recover known formulas.
  • Power system data: Industrial data from the Grid2Op platform is used to represent historical sensor measurements, posing a more complex exploratory SR task.

Key Insights and Ideas

  • Human-in-the-loop SR is crucial for interpretability and exploration: When the true formula is unknown, human intuition and domain knowledge are invaluable for guiding the search towards meaningful expressions.
  • Multi-format preference elicitation is key to user engagement: Offering diverse ways for users to provide feedback (categorizing, comparing, suggesting) makes the interaction less tedious and more effective. This caters to different user preferences and expertise levels.
  • Grammar-guided generation combined with preferences leads to more relevant solutions: The grammar ensures valid expressions, while preferences steer the RL agent towards human-validated results, bridging the gap between algorithmic exploration and human understanding.
  • Interactive SR can be applied to complex industrial problems: The power system example demonstrates the potential of this approach for uncovering hidden relationships in real-world data, where interpretability is paramount.

Key Results

The paper focuses on the interactive platform and its methodology rather than presenting extensive quantitative results on specific benchmarks. The key "results" are the successful implementation and demonstration of the interactive platform.

  • The platform allows users to guide SR through multiple preference feedback mechanisms.
  • It has been tested on benchmark functions and a power system dataset.
  • The system aims to reduce user boredom and increase engagement by minimizing the number of required interactions.

Limitations and Future Work

  • User Expertise: The "Expression suggestion" tab is noted as being more suited for users familiar with BNF grammars, suggesting a potential barrier for less technical users.
  • Interaction Fatigue: While the paper aims to mitigate this, the authors acknowledge that interaction schemes can be "exhausting" or "annoying" for users, implying that further optimization of the interaction process might be needed.
  • Future Work: The authors suggest testing and comparing the different interaction modalities, both individually and jointly, to understand their effectiveness. They also highlight the potential for this preference-based learning approach in exploratory SR tasks where users have insights but not the exact solution.

Summary updated

BibTeX
@inproceedings{crochepierre2022interactive,
  author = {Laure Crochepierre and Lydia Boudjeloud-Assala and Vincent Barbesant},
  title = {{Interactive Reinforcement Learning for Symbolic Regression from Multi-Format Human-Preference Feedbacks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/849},
}
Figure