System identification—the art of building mathematical models from raw data—has long been the domain of human experts. It requires a delicate balance of selecting the right model structure, embedding physical laws, and rigorous validation. While automated tools like Symbolic Regression (SR) exist, they often struggle to incorporate "common sense" physics or explain why a specific equation was chosen.
A new paper, "ASPIRE: An Agentic System-Identification Platform for Iterative Reasoning and Exploration," introduces a framework that puts Large Language Model (LLM) agents in the driver's seat of this process. ASPIRE doesn't just crunch numbers; it reasons about the system, writes its own identification code, and documents its findings just like a human researcher would.
TL;DR
ASPIRE is an agentic framework where LLMs autonomously select, implement, and execute system identification methods to create interpretable, closed-form models. By combining a "harness" for rigorous verification with the reasoning power of LLMs, it produces models that are not only accurate but also physically consistent and fully documented.
What Problem Does This Paper Address?
Traditional system identification is a "white-box" modeling challenge. Engineers prefer closed-form equations (like $F=ma$) over "black-box" neural networks because equations are interpretable, easier to embed in controllers, and more reliable when extrapolating beyond training data.
However, automating this is hard. Existing tools like Sparse Identification of Nonlinear Dynamics (SINDy) or standard SR packages are often "fixed-pipeline" tools—they follow a set algorithm and don't "understand" the physics of the system they are modeling. Furthermore, recent attempts to use LLMs for this task often suffer from test leakage (where the model accidentally sees the test data) or a lack of transparency in how the final equation was derived.
Methods
ASPIRE splits the workload between two main components: the Agent and the Harness.
- The Agent (The "Brain"): An LLM (like GPT-4 or similar) that receives a task brief. It can inspect training data, search the web for relevant physics, propose model structures, and write Python code to fit parameters. It can even start "helper agents" to parallelize tasks.
- The Harness (The "Guardrails"): This is a software environment that executes the agent's commands. Crucially, it keeps the test data sealed—the agent never sees it. The harness also performs independent verification of formal physical requirements (e.g., ensuring a robot's heading follows kinematic laws).
The process is iterative. In interactive mode, the agent can ask the user clarifying questions (e.g., "Is there a delay in the sensor?"), and the user can provide feedback to refine the model pool. ASPIRE also enforces a strict documentation format, requiring the agent to explain every term in the equation and justify its modeling choices.
Datasets
The authors evaluated ASPIRE on three real-world experimental systems, testing each in both "described" (with physics context) and "blind" (data-only) modes:
- Mobile Robot: A TurtleBot 2 where the goal is to model position and heading based on velocity commands.
- Friction System: A rotational disc influenced by gravity and dry friction (stick-slip behavior).
- Fermenter: A laboratory tank where the agent must model air pressure dynamics based on valve positions and flow rates.
Key Insights and Ideas
- Physics over Pure Accuracy: Interestingly, providing a physical description didn't always lower the test error, but it significantly improved the model quality. Described models had fewer parameters and used physically meaningful terms (like sines for gravity torque) rather than arbitrary high-degree polynomials.
- The "Sealed Store" is Essential: By preventing the agent from accessing test data, ASPIRE provides a true measure of generalization. This addresses a major criticism of previous agentic SysID papers where "implicit leakage" was a concern.
- Agents as Tool-Builders: Unlike previous systems that give LLMs a fixed toolbox (like "use PySR"), ASPIRE agents often wrote their own custom least-squares fitting scripts or implemented specific physics-based structures from scratch.
- Verification as a Filter: The harness's ability to symbolically or numerically verify constraints (like monotonicity) allowed it to reject models that were mathematically "accurate" on training data but physically impossible.
Key Results
- Performance: ASPIRE models consistently outperformed or matched established tools like PySINDy and PySR in predictive accuracy (RMSE).
- Physical Consistency: In the friction system, standard tools frequently violated basic physical requirements (like the direction of gravity). ASPIRE models, guided by formal requirements, satisfied all of them.
- Reliability: In 30 autonomous runs, the agent delivered valid models every single time.
- Cost: The average cost per run was remarkably low, ranging from $0.96 to $3.38, making it a highly accessible tool for researchers.
- Documentation Quality: An expert assessment found that the agent's explanations and documentation fully satisfied 132 out of 150 assessed criteria.
Limitations and Future Work
While impressive, the authors note several boundaries for the current version of ASPIRE:
- Single Output: The current framework is restricted to systems with a single scored output.
- Measured States: It assumes all system states are directly measured (no hidden states requiring observers).
- Human-in-the-loop: While it can run autonomously, the highest quality models often came from the interactive mode where a human could correct the agent's assumptions.
Future Work: The team plans to extend ASPIRE to multi-output systems, handle unmeasured states, and even have the agent guide data collection by suggesting which experiments the user should run next to resolve model ambiguities.
Summary updated