Learning Concept Abstractness Using Weak Supervision

Ella Rabinovich, Benjamin Sznajder, Artem Spector, Ilya Shnayderman, Ranit Aharonov, David Konopnicki, Noam Slonim

Abstract

Distinguishing between abstract concepts (e.g., "justice") and concrete ones (e.g., "apple") is a fundamental task in psycholinguistics, yet scaling this to large vocabularies is limited by the cost of human annotation. This paper introduces a weak supervision framework that utilizes morphological rules and contextual patterns to learn abstractness. By training on "noisy" labels derived from word suffixes and Wikipedia context, an RNN-based model achieves near-human correlation with expert ratings, demonstrating high data efficiency and generalizability across various datasets.

Introduction

In psycholinguistics, "abstractness" refers to the degree to which a concept cannot be directly perceived through the five senses. While intuitive for humans, this distinction presents a significant challenge for AI systems. Existing resources for word abstractness are often small due to the reliance on manual labeling, which creates a bottleneck for downstream Natural Language Processing (NLP) tasks such as metaphor detection, lexical simplification, and readability assessment.

This research addresses the data scarcity problem by demonstrating that abstractness can be inferred from linguistic structure and context without massive human-labeled datasets. The authors propose a "weak supervision" approach that bootstraps learning from simple morphological cues.

Methodology

The proposed approach integrates two primary linguistic signals: Morphology and Context.

Weak Labeling and Seed Rules

Instead of human-provided labels, the researchers utilized a "seed" rule based on morphology. Words with specific suffixes, such as -ism (e.g., marxism) and -ness (e.g., happiness), were automatically labeled as abstract. These were paired with a set of random words assumed to be concrete, forming a noisy training set.

Contextual Learning

The models were trained on 800,000 sentences extracted from Wikipedia. To prevent the models from simply memorizing specific words, the target concepts were masked during training. This forced the models to identify abstractness based solely on the surrounding linguistic environment—the "company a word keeps."

Model Architectures

Three distinct modeling approaches were evaluated:

  1. Naive Bayes (NB): A baseline probabilistic model utilizing word counts.
  2. Nearest Neighbor (NN-RAD): A model using GloVe word embeddings to identify similar concepts in vector space.
  3. RNN with Attention: A Bidirectional LSTM equipped with an attention mechanism to prioritize contextual words that best indicate abstractness.

Experimental Setup

The models were trained on 1,040 unigrams from Wikipedia and evaluated against several "gold standard" benchmarks:

  • MRC Database: A standard psycholinguistic resource.
  • BWK Dataset: A large-scale manual annotation containing 40,000 concepts.
  • New Evaluation Set: A custom dataset of 1,500 Wikipedia concepts (unigrams, bigrams, and trigrams) annotated by seven human experts.

Results and Discussion

The RNN model emerged as the superior approach, particularly for unigrams, achieving a 0.74 Pearson correlation with human labels. This performance is notable as it falls within the range of typical human-to-human agreement (0.65–0.72).

Comparison of Pearson correlation scores across different test sets and n-gram lengths.

Data Efficiency and Generalization

A key finding was the model's remarkable data efficiency. As shown in the performance analysis below, the RNN model achieves high correlation with human intuition using as few as 10 sentences per concept.

Model performance (Pearson correlation) as a function of the number of sentences used for training.

Beyond performance, the model demonstrated an ability to "discover" new morphological markers. After training, it successfully identified other suffixes associated with abstractness, such as -sis (e.g., analysis) and -ics (e.g., politics), which were not part of the initial seed rules.

Complexity and Limitations

While the model excels at unigrams, performance decreases as concept complexity increases. Trigrams (e.g., "absolute monarchy") achieved a correlation of 0.49, likely due to the higher subjectivity and lower human agreement associated with complex phrases. Additionally, the use of static embeddings limits the model's ability to handle polysemy, where a word like "bank" may have both concrete and abstract senses depending on context.

Conclusion

This work demonstrates that weak supervision is a powerful tool for capturing nuanced psychological properties like abstractness. By combining simple morphological heuristics with contextual deep learning, the authors provide a scalable "recipe" for building psycholinguistic resources in a data-efficient manner. This approach holds significant promise for adaptation to other languages where labeled data is currently unavailable.

Summary updated

Abstract

We introduce a weakly supervised approach for inferring the property of abstractness of words and expressions in the complete absence of labeled data. Exploiting only minimal linguistic clues and the contextual usage of a concept as manifested in textual data, we train sufficiently powerful classifiers, obtaining high correlation with human labels. The results imply the applicability of this approach to additional properties of concepts, additional languages, and resource-scarce scenarios.
BibTeX
@inproceedings{rabinovich2018learning,
  author = {Ella Rabinovich and Benjamin Sznajder and Artem Spector and Ilya Shnayderman and Ranit Aharonov and David Konopnicki and Noam Slonim},
  title = {{Learning Concept Abstractness Using Weak Supervision}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1522},
}
Figure