Joint Concept Learning and Semantic Parsing from Natural Language Explanations

Shashank Srivastava, Igor Labutov, Tom M. MitchellEMNLP 2017 · OpenReview

Introduction

Traditional machine learning models are notoriously "data-hungry," often requiring thousands of labeled examples to learn a simple concept like identifying a phishing email. This stands in stark contrast to human learning, where a single natural language explanation—such as "It’s a phishing email if it asks for your bank account number"—can be immediately understood and applied.

Researchers at Carnegie Mellon University have developed a framework to bridge this gap. By jointly training a semantic parser and a classifier, their system learns new concepts from natural language explanations and a minimal number of labeled examples. This approach addresses the "long tail" of user-specific concepts (e.g., "emails from my postdoc applicants") where massive pre-labeled datasets are unavailable.

The Joint Learning Framework

The core of the proposed approach is a joint model that learns to parse natural language and classify data simultaneously using a Generalized Expectation-Maximization (EM) procedure.

The joint learning framework architecture, showing the feedback loop between the semantic parser and the classifier.

As illustrated in Figure 1, the system consists of several interconnected components:

  • The Semantic Parser: Maps natural language statements to logical expressions using a Combinatory Categorial Grammar (CCG).
  • The Concept Classifier: Uses the output of the parsed explanations as binary features to categorize data.
  • Latent Variable Learning: The model treats the "truth value" of an explanation for a specific instance as a latent variable. During training, it favors interpretations that are both linguistically plausible and discriminative (i.e., they help distinguish between classes).

Crucially, this requires no labeled logical forms. The system learns what a phrase like "subject contains" means simply because that interpretation helps it correctly label the training data.

Data Collection

Because no existing datasets paired emails with human explanations, the authors created a new corpus. They collected 1,030 emails across seven concepts, such as Meeting Request and Policy Announcement. Using Amazon Mechanical Turk, they gathered 235 natural language statements through "Teaching" tasks, where users wrote rules to explain how to identify specific email categories.

Experimental Results

The model was evaluated in a "low-data regime," simulating scenarios where a user provides only a few examples. The results demonstrate that utility serves as a powerful proxy for meaning: if a specific interpretation of a sentence helps a classifier achieve high accuracy, that interpretation is likely correct.

Performance comparison showing Macro Average F1 scores for the LNL model versus a Bag-of-Words (BoW) baseline across varying numbers of labeled examples.

As shown in Figure 2, the Learning from Natural Language (LNL) approach significantly outperforms the Bag-of-Words (BoW) baseline. In the most data-constrained scenarios (10 labeled examples per concept), the joint model achieved a 30% relative improvement in F1 score. Furthermore, despite having zero labeled logical forms for training, the parser reached an exact-match accuracy of approximately 30%, which was sufficient to provide a substantial boost to classification performance.

Discussion and Future Work

While the LNL framework shows great promise, several hurdles remain. The system currently relies on a domain-specific lexicon and struggles with highly complex language—about 25% of human explanations were too complex for the current logical language.

Future work aims to incorporate linguistic nuances like modifiers (e.g., "usually" or "rarely") and develop mixed-initiative dialogs. In such a system, the machine could detect when an explanation contains an unknown concept and proactively ask the user to "teach" that sub-concept recursively, moving closer to a truly natural pedagogical interaction between humans and machines.

Summary updated

BibTeX
@inproceedings{srivastava2017joint,
  author = {Shashank Srivastava and Igor Labutov and Tom M. Mitchell},
  title = {{Joint Concept Learning and Semantic Parsing from Natural Language Explanations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=r1ZxvfMuZr},
}
Figure