AutoMATES: Automated Model Assembly from Text, Equations, and Software

Adarsh Pyarelal, Marco Antonio Valenzuela-Escárcega, Rebecca Sharp, Paul Douglas Hein, Jon Stephens, Pratik Bhandari, HeuiChan Lim, Saumya Debray, Clayton T. MorrisonarXiv 2020 · OpenReview

TL;DR

This paper introduces AutoMATES, a system designed to automatically extract, unify, and represent scientific models from disparate sources like natural language text, equations, and source code. This unified representation aims to break down silos between different modeling domains, enabling easier integration of complex systems and facilitating automated model composition.

What Problem Does This Paper Address?

Scientists often model complex phenomena using multiple, specialized computational models. Integrating these models to study overarching problems (like food security, which involves crop, weather, and hydrology models) is a significant challenge. This integration is hindered by the fact that models are represented in various forms – natural language descriptions in publications, mathematical equations, and source code – with no common, semantically rich language to bridge them. AutoMATES aims to solve this by creating a unified, semantically rich representation that can link these different forms of model information.

Methods

The AutoMATES system has a modular architecture with four main components:

  1. Extraction: This phase pulls information from three sources:
  2. Source Code: Analyzes code (using a tool called for2py for Fortran) and comments to extract model logic and descriptions.
  3. Text: Employs a machine reading framework (based on Eidos) to extract model variables and their descriptions from scientific publications.
  4. Equations: Uses computer vision techniques to locate equations in PDFs, converts them to LaTeX, and then parses them into an intermediate symbolic representation.
  5. Grounding: This step links concepts across the different extracted sources. It identifies when the same model variable or concept is expressed in text, equations, or code, creating a unified, language-agnostic representation called a Grounded Function Network (GrFN).
  6. Comparison: Models are compared using their GrFN representations, analyzing structural and functional similarities and differences. This includes sensitivity analysis to understand how variables influence each other.
  7. Augmentation: This phase uses the unified representations to compose models for specific tasks, generate descriptions, and facilitate model execution.

Datasets

The paper does not explicitly list specific datasets used for training or evaluation. However, it mentions using scientific papers (PDFs) and corresponding source code as the primary input for the system. The examples shown are derived from crop modeling systems like DSSAT.

Key Insights and Ideas

  • Unified Representation is Key: The core idea is that a common, semantically rich representation (GrFN) is crucial for integrating models from diverse sources and domains.
  • Leveraging Multiple Modalities: The system's strength lies in its ability to process and connect information from text, equations, and code, recognizing that each provides complementary insights into a model.
  • Automated Model Curation: AutoMATES promises to significantly reduce the manual effort currently required to curate and integrate scientific models, accelerating research in complex systems.
  • Bridging the Gap: By grounding extracted information, the system can link abstract mathematical concepts to concrete code implementations and their textual explanations, making models more understandable and usable.

Key Results

The paper presents qualitative results and examples of the system's functionality:

  • Figure 1 illustrates the integration of text, equations, and code into a semantically-enriched Bayesian network.
  • Figure 3 shows an example of variables extracted from scientific text, represented as concepts, definitions, and value assignments.
  • Figure 4 visualizes the comparison between two evapotranspiration models (Priestley-Taylor and ASCE), highlighting shared and unique variables.
  • Figure 5 is a screenshot of the AutoMATES CodeExplorer, demonstrating the translation of a Fortran subroutine into a computation graph with annotated variables.
  • Figure 6 shows initial results of automated sensitivity analysis, identifying key variables influencing potential evapotranspiration.

Quantitative results regarding accuracy or performance metrics are not detailed in this paper.

Limitations and Future Work

  • Scalability: The paper acknowledges the challenge of scalability for very large software systems and mentions performing analysis at the module level to address this.
  • Fortran Subset: The for2py tool currently handles a significant subset of Fortran, but the handling of modules and derived types is still being completed.
  • Equation Parsing: While promising, the evaluation of the equation parsing process on a held-out subset of ArXiv data is ongoing.
  • Future Work: The authors aim to extend the framework to support more knowledge sources and domains, enabling the modeling of large, complicated systems at multiple levels of abstraction. They are also developing methods for automatic code differentiation and Bayesian optimization to improve the efficiency of sensitivity analysis.

Summary updated

Abstract

Models of complicated systems can be represented in different ways - in scientific papers, they are represented using natural language text as well as equations. But to be of real use, they must also be implemented as software, thus making code a third form of representing models. We introduce the AutoMATES project, which aims to build semantically-rich unified representations of models from scientific code and publications to facilitate the integration of computational models from different domains and allow for modeling large, complicated systems that span multiple domains and levels of abstraction.
BibTeX
@misc{pyarelal2020automates,
  author = {Adarsh Pyarelal and Marco Antonio Valenzuela-Escárcega and Rebecca Sharp and Paul Douglas Hein and Jon Stephens and Pratik Bhandari and HeuiChan Lim and Saumya Debray and Clayton T. Morrison},
  title = {{AutoMATES: Automated Model Assembly from Text, Equations, and Software}},
  year = {2020},
  eprint = {2001.07295},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.07295},
}
Figure