RELATIONS - Workshop on meaning relations between phrases and sentences
María Becker, Michael Staniek, Vivi Năstase, Anette Frank, Nina Khairova, Svitlana Petrasova, Kuralay Mukhsina, Frieda Josi, Christian Wartena, Ulrich Heid, Mohammed Attia, Suraj Maharjan, Younes Samih, Laura Kallmeyer, Thamar Solorio, Huihui He, Rui Xia, Iris Hendrickx, Nam Su, Zornitsa Kim, Preslav Kozareva, Nakov, Diarmuid, Sebastian Saghdha, Marco Pad, Lorenza Pennacchiotti, Stan Romano, Szpakowicz, Jos-Angel Gonzlez, Hurtado Oliver, Llus Segarra, Encarna Pla, Ferran, Diarmuid Seaghdha, Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, Manfred Pinkal, Adam Paszke, Sam Gross, Adam Lerer, Jeffrey Pennington, Richard Socher, Christopher Manning, Feiliang Ren, Di Zhou, Zhihui Liu, Yongcheng Li, Rongsheng Zhao, Yongkang Liu, Xiaobo Liang, Itsumi Saito, Kyosuke Nishida, Hisako Asano, Junji Tomita, Seiya Shudo, Rafal Rzepka, Kenji Araki, Robert Speer, Joshua Chin, Catherine Havasi, Henry Lieberman, Zhen Tan, Bo Li, Peixin Huang, Bin Ge, Weidong Xiao, Ekaterina Vylomova, Laura Rimell, Trevor Cohn, Timothy Baldwin, Zhigang Wang, Juanzi Li, Dirk Weissenborn, Tomas Kocisky, Chris Dyer, I Akermi, R Faiz, C Boom, S Canneyt, S Bohez, T Demeester, B Dhoedt, A Dang, R Makki, A Moh'd, A Islam, V Keselj, E Milios, J Ganitkevitch, B Durme, C Callison-Burch, L Han, T FininOpenalex Unknown2019 · DOI · OpenAlex
Commonsense knowledge relations are fundamental to advanced Natural Language Understanding (NLU). This study examines the learnability of relations in ConceptNet, focusing on challenges such as multi-label ambiguity and multi-word arguments. We propose a neural open-world multi-label classification approach, utilizing relation-specific thresholds and Bi-LSTM encoders. Our findings highlight that argument complexity and relation overlap are primary obstacles, while also addressing the inherent incompleteness of the ConceptNet resource.
Introduction: The Hurdles of Commonsense NLU
While large-scale knowledge graphs like ConceptNet are vital for NLU, they present significant challenges for automated systems. The researchers identify three primary hurdles:
Non-Disjoint Relations: Concept pairs often share multiple relationships (e.g., a pair could be linked by both Desires and CapableOf), unlike standard datasets that assume a single label per instance.
Argument Complexity: Concepts are frequently multi-word expressions (e.g., "playing a harp") rather than single tokens, requiring sophisticated encoding to capture semantic nuances.
Resource Incompleteness: ConceptNet is an evolving resource. A model's "false positive" prediction might actually be a valid relation missing from the database, complicating standard evaluation metrics.
Methodology: A Multi-Label Neural Approach
The study employs a multi-label neural relational classifier designed to handle the messy reality of commonsense data:
Input Encoding: The team compared simple centroid vectors (averaging word embeddings) against Bi-LSTMs to capture the structure of multi-word concepts.
Architecture: A Feed-Forward Neural Network (FFNN) with a single hidden layer and a sigmoid output layer to treat each relation as an independent binary classification task.
Relation-Specific Thresholding: Instead of a global cutoff, the model tunes individual probability thresholds for each of the 14 target relation types, which significantly improves F1 scores for difficult relations.
Open-World Setup: The inclusion of "RANDOM" and "OTHER" classes tests the model's ability to identify when concepts have no relationship or one outside the target set.
Data and Evaluation
The research focused on the English subpart of the Open Mind Common Sense (OMCS) project. The authors curated CN-OMCS-14, a balanced dataset of the 14 most frequent relations (e.g., UsedFor, HasProperty, AtLocation). Evaluation utilized a 10/10/80 split, with instances sorted by ConceptNet’s internal confidence scores to ensure a high-quality gold standard for testing.
Key Insights and Results
The Ambiguity Tax: There is a strong inverse correlation (-0.82) between a relation's performance and its semantic overlap with others. Relations like HasSubevent and HasPrerequisite remain difficult due to high ambiguity.
RNN Superiority: Bi-LSTM encoders consistently outperformed simple vector averaging, particularly for relations involving complex actions and events.
The False Positive Paradox: Manual annotation revealed that 9% of the model's "errors" were actually valid relations missing from ConceptNet, suggesting that current benchmarks underestimate true model performance.
Performance Summary
The best-performing model (FFNN + RNN with tuned thresholds) achieved:
Closed World F1: 0.71
Open World F1: 0.65 - 0.68
Top Performers:Desires (0.94 F1) and ReceivesAction (0.91 F1).
Bottom Performers:HasSubevent (0.24 F1) and HasPrerequisite (0.38 F1).
Conclusion and Future Directions
The study underscores that the incompleteness of ConceptNet is a major bottleneck for evaluation. Future work will focus on developing meta-classifiers to learn relation co-occurrence patterns and using these neural models to actively enrich ConceptNet by predicting and verifying new knowledge edges.
Summary updated
Abstract
Commonsense knowledge relations are crucial for advanced NLU tasks.We examine the learnability of such relations as represented in CONCEPTNET, taking into account their specific properties, which can make relation classification difficult: a given concept pair can be linked by multiple relation types, and relations can have multi-word arguments of diverse semantic types.We explore a neural open world multi-label classification approach that focuses on the evaluation of classification accuracy for individual relations.Based on an in-depth study of the specific properties of the CONCEPTNET resource, we investigate the impact of different relation representations and model variations.Our analysis reveals that the complexity of argument types and relation ambiguity are the most important challenges to address.We design a customized evaluation method to address the incompleteness of the resource that can be expanded in future work.(iii) As any knowledge resource, CONCEPTNET is incomplete, which means that relations between concepts are missing.The incompleteness of the resource poses serious evaluation problems, since assumed negative instances may in fact be positive.To tackle these issues we perform a thorough experimental examination of the learnability of CON-CEPTNET relations in a controlled multi-label classification setting.Our contributions are: (i) a cleaned and balanced data subset covering the 14 most frequent relation types from the core part of CONCEPT-NET that serves as a basis for assessing relation-specific classification performance.We extend this dataset to an open-world classification setup; (ii) a neural multi-label classification approach with various model options for the representation of relations and their (multi-word) arguments, including relationspecific label prediction thresholds; (iii) an in-depth analysis of specific properties of the CONCEPT-NET relation inventory, from which we derive hypotheses that we evaluate in classification experiments; (iv) we perform detailed analysis of results that confirm a great number of our hypotheses regarding specific classification challenges; (v) finally, we assess the amount of potential evaluation discrepancies due to the incompleteness of the resource in a small-scale annotation experiment.2 Related Work Semantic Relation ClassificationSemantic relation classification covers a wide range of methods and learning paradigms for representing relation instances (see Nastase et al. 2013 for an overview).Typically, the data is presented to the learner as independent instances, with or without a sentential context.Relation classification models represent the meaning of the arguments (attributional features) and if context is available, also the relation (relational features).Recently Deep Learning has strongly influenced semantic relation learning.Word embeddings can provide attributional features for a variety of learning frameworks (Attia et al., 2016;Vylomova et al., 2016), and the sentential context -in its entirety, or only the structured (through grammatical relations) or unstructured phrase expressing the relation -can be modeled through a variety of neural architectures -CNN (Tan et al.,
BibTeX
@inproceedings{becker2019relations,
author = {María Becker and Michael Staniek and Vivi Năstase and Anette Frank and Nina Khairova and Svitlana Petrasova and Kuralay Mukhsina and Frieda Josi and Christian Wartena and Ulrich Heid and Mohammed Attia and Suraj Maharjan and Younes Samih and Laura Kallmeyer and Thamar Solorio and Huihui He and Rui Xia and Iris Hendrickx and Nam Su and Zornitsa Kim and Preslav Kozareva and Nakov and Diarmuid and Sebastian Saghdha and Marco Pad and Lorenza Pennacchiotti and Stan Romano and Szpakowicz and Jos-Angel Gonzlez and Hurtado Oliver and Llus Segarra and Encarna Pla and Ferran and Vivi Nastase and Preslav Nakov and Diarmuid Seaghdha and Stan Szpakowicz and Simon Ostermann and Michael Roth and Ashutosh Modi and Stefan Thater and Manfred Pinkal and Adam Paszke and Sam Gross and Adam Lerer and Jeffrey Pennington and Richard Socher and Christopher Manning and Feiliang Ren and Di Zhou and Zhihui Liu and Yongcheng Li and Rongsheng Zhao and Yongkang Liu and Xiaobo Liang and Itsumi Saito and Kyosuke Nishida and Hisako Asano and Junji Tomita and Seiya Shudo and Rafal Rzepka and Kenji Araki and Robert Speer and Joshua Chin and Catherine Havasi and Robert Speer and Catherine Havasi and Henry Lieberman and Zhen Tan and Bo Li and Peixin Huang and Bin Ge and Weidong Xiao and Ekaterina Vylomova and Laura Rimell and Trevor Cohn and Timothy Baldwin and Zhigang Wang and Juanzi Li and Dirk Weissenborn and Tomas Kocisky and Chris Dyer and I Akermi and R Faiz and C Boom and S Canneyt and S Bohez and T Demeester and B Dhoedt and A Dang and R Makki and A Moh'd and A Islam and V Keselj and E Milios and J Ganitkevitch and B Durme and C Callison-Burch and L Han and T Finin},
title = {{RELATIONS - Workshop on meaning relations between phrases and sentences}},
booktitle = {Openalex Unknown},
year = {2019},
doi = {10.18653/v1/w19-08},
}