Learning Pipelines with Limited Data and Domain Knowledge: A Study in Parsing Physics Problems

Mrinmaya Sachan, Kumar Avinava Dubey, Tom M. Mitchell, Dan Roth, Eric P. Xing

Modern machine learning faces a significant "data-hungriness" problem. While end-to-end (E2E) models achieve state-of-the-art results in general domains, they require massive datasets that simply don't exist for niche, complex tasks like parsing textbook physics problems.

Traditional pipelines offer a workaround: they break a complex task into smaller, manageable sub-tasks (e.g., first detect lines, then arrows, then forces). This modularity makes it easy to plug in human-engineered rules or pre-existing software. However, pipelines suffer from error propagation. If the very first stage makes a mistake, that error cascades through every subsequent stage, often leading to a total failure.

To bridge this gap, researchers introduced Nuts&Bolts, a framework that combines the modularity and knowledge-integration of a pipeline with the global optimization and error-resilience of an E2E model.

An example physics problem and its parsed formal representation.

The Nuts&Bolts Framework

The Nuts&Bolts framework is built on Probabilistic Soft Logic (PSL). It tackles the brittleness of traditional pipelines through several key technical innovations:

  • Joint Modeling: Instead of running stages sequentially, the entire pipeline is represented as a single probabilistic logic program. This allows information to flow bidirectionally. For example, knowing that a text label says "Force" can help the system go back and correctly identify a previously ambiguous line as an arrow.
  • Trustworthiness Modeling: In many pipeline stages, there are multiple ways to solve a problem (e.g., using three different corner detectors). Nuts&Bolts assigns a "trust" variable to each function approximator, learning which "expert" to rely on based on how well their outputs agree with the rest of the system.
  • Integration of Rules and Software: Domain knowledge (like "a pulley is a circle with two tangent lines") is encoded as logical implications. These act as "soft" constraints that guide the model, especially when labeled data is scarce.
  • Semi-Supervised Learning: Because the framework is probabilistic, it can incorporate unlabeled data by maximizing the consistency of its internal logic across thousands of examples for which no "ground truth" answer is known.
The modular pipeline architecture for parsing physics diagrams, which Nuts&Bolts jointly optimizes.

Datasets and Evaluation

The primary evaluation uses a newly curated dataset of 4,941 Newtonian physics questions extracted from popular pre-university textbooks. For diagram parsing, 1,019 problems had associated diagrams, with a subset annotated for training and testing. The framework was compared against computer vision baselines like G-ALIGNER and DSDP-Net.

To prove the framework generalizes beyond physics, the authors also tested it on four standard Relation Extraction datasets (CDR, Spouses, KBP News, and Genomics), comparing it against the popular weak-supervision framework Snorkel.

Key Insights and Results

  • The Power of Joint Inference: The most significant insight is that joint training effectively "cures" the pipeline's biggest weakness. By allowing later stages to influence earlier ones, the system becomes robust to local noise.
  • Unlabeled Data as a Force Multiplier: In niche domains, unlabeled data is often "free." Adding 912 unlabeled diagrams improved the parsing F1 score from 0.58 to 0.74—a massive jump highlighting the power of semi-supervised logic.
  • Rules as Regularizers: Human rules function as a form of structured regularization, preventing the model from making physically impossible predictions when training data is sparse.
  • Performance Leaps: Nuts&Bolts achieved a 0.74 F1 score and 0.78 Jaccard similarity for diagram parsing, significantly outperforming the best baseline (G-ALIGNER at 0.65 F1). When plugged into a physics solver, the system improved question-answering accuracy from ~21% to 32%.
  • Cross-Domain Success: In the Relation Extraction task, Nuts&Bolts consistently outperformed Snorkel, suggesting that modeling the "trustworthiness" of different labeling functions is superior to Snorkel's generative modeling approach.

Limitations and Future Work

While highly effective, Nuts&Bolts still requires manual rule engineering by domain experts to define concepts like a "wedge" or "ground." Additionally, while PSL uses relatively fast convex optimization, very large, deeply nested pipelines might still face scaling challenges. Finally, the authors noted that highly subjective natural language in some physics exams remains difficult to parse, indicating a need for better NLP integration in future iterations.

Summary updated

Abstract

As machine learning becomes more widely used in practice, we need new methods to build complex intelligent systems that integrate learning with existing software, and with domain knowledge encoded as rules. As a case study, we present such a system that learns to parse Newtonian physics problems in textbooks. This system, Nuts&Bolts, learns a pipeline process that incorporates existing code, pre-learned machine learning models, and human engineered rules. It jointly trains the entire pipeline to prevent propagation of errors, using a combination of labelled and unlabelled data. Our approach achieves a good performance on the parsing task, outperforming the simple pipeline and its variants. Finally, we also show how Nuts&Bolts can be used to achieve improvements on a relation extraction task and on the end task of answering Newtonian physics problems.
BibTeX
@inproceedings{sachan2018learning,
  author = {Mrinmaya Sachan and Kumar Avinava Dubey and Tom M. Mitchell and Dan Roth and Eric P. Xing},
  title = {{Learning Pipelines with Limited Data and Domain Knowledge: A Study in Parsing Physics Problems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
}
Figure