CLaC-np at SemEval-2021 Task 8: Dependency DGCNN

Nihatha Lathiff, Pavel Pk Khloponin, Sabine BerglerACL 2021

TL;DR

This paper tackles the challenging task of extracting quantities and their associated measured entities and properties from scientific texts. They propose a novel approach using Deep Graph Convolutional Networks (DGCNN) that leverages dependency parse trees to capture rich structural information, achieving competitive results on the SemEval-2021 Task 8 benchmark.

What Problem Does This Paper Address?

Scientific literature is rich with quantitative information, but extracting this data accurately is surprisingly difficult. Simply identifying numbers isn't enough; understanding what is being measured (the entity) and how it's being measured (properties or qualifiers) is crucial. This is particularly challenging due to the diverse writing styles and complex sentence structures found in scientific documents. SemEval-2021 Task 8, "MeasEval," specifically aims to address this by requiring systems to identify quantity spans, their units, measured entities, properties, and qualifiers, and the relationships between them. Previous work often under-reports the impact of data preprocessing and postprocessing steps, which can significantly influence performance.

Methods

The authors propose a three-phase system:

  • Phase I: Preprocessing: This phase focuses on preparing the text for analysis. Key steps include:
  • Specialized Tokenization: Custom rules are developed to handle mixed character types (e.g., "813СТОС"), mathematical symbols (e.g., "5 < 20/° < 80"), number normalization, and abbreviations (e.g., "e.g.", "Fig.").
  • List and Interval Protection: Manual assignment of "CD" (cardinal number) POS tags to groupings of numbers in lists and intervals to improve dependency parsing.
  • Unit Gazetteer: A comprehensive list of units is compiled from various sources.
  • Phase II: Input Creation and GCN Training: This is the core of their approach.
  • DGCNN Architecture: They employ a Deep Graph Convolutional Neural Network (DGCNN) to process dependency parse trees as graphs. This allows the model to capture higher-order neighborhood information and structural relationships.
  • Graph Representation: The dependency tree is modeled as a graph (V, E), where V are tokens and E are directed dependency relations. Explicit reverse links are added to capture bidirectional relationships.
  • Feature Engineering: For each node in the graph, they construct an information matrix (I) using several features:
  • Distance Feature: A one-hot vector indicating the distance to the subgraph center nodes (t1 and t2).
  • POS Feature: A one-hot vector representing the Part-of-Speech tag.
  • Word Embedding: Pre-trained ELMo embeddings.
  • Dependency Path Embedding: Embeddings derived from dependency paths using node2vec, trained on dependency sequences rather than just nodes.
  • UMAP Embedding: Dimensionality reduction applied to dependency embeddings.
  • Subgraph Extraction: For candidate quantity tokens (t1), they extract a subgraph centered around the shortest path to a potential related token (t2).
  • Phase III: Postprocessing: This phase bridges the gap between token-level predictions and the competition's span-level output.
  • Class Labelling: The DGCNN predicts one of six classes representing the relationship between token pairs (t1, t2).
  • Prediction Ranking: If multiple t2 tokens are predicted for a single t1, the one with the highest probability is chosen.
  • Span Mapping: Token-level predictions are converted into spans.
  • Unit Detection: A gazetteer is used to identify units within quantity spans.
  • Quantity Modifier Prediction: A pre-trained BERT model is used to predict modifiers for quantity spans.

Datasets

The paper uses the MeasEval 2021 Task 8 dataset for training and evaluation. They split the combined training and trial datasets into 75% for training and 25% for validation.

Key Insights and Ideas

  • Dependency Structure is Key: Encoding dependency parse trees as graphs and using DGCNNs proves effective for capturing the structural relationships crucial for linking quantities to their context.
  • Preprocessing Matters: Specialized tokenization rules and handling of lists/intervals significantly impact performance, highlighting the importance of domain-specific text preparation.
  • Node2vec on Dependency Paths: Applying node2vec to dependency sequences (rather than just nodes) to generate embeddings for dependency relationships is a novel way to encode structural information.
  • UMAP's Mixed Results: While UMAP showed a slight improvement on validation data, it led to overfitting and performance degradation on the test set, suggesting it's not a universally beneficial addition for this task.
  • Locality is Important: Larger subgraphs and longer dependency paths tend to degrade performance, indicating that task-oriented locality features are beneficial.

Key Results

  • The system achieved competitive results on the SemEval-2021 Task 8, particularly for Quantity (Q) and Unit (U) extraction, which were in line with stronger systems.
  • Performance for Measured Entities (ME) and Measured Properties (MP) was lower, suggesting that the multi-class labeling approach needs further refinement.
  • Ablation studies showed that including list and interval protection improved results, though a high rate of duplicates was observed initially.
  • UMAP reduction provided a small boost on validation data but a performance loss on test data.

Limitations and Future Work

  • Data Size: The limited size of the dataset restricts the ability to draw very general conclusions.
  • ME/MP Performance: The lower performance on measured entities and properties indicates a need for better support for these specific tasks. The authors suggest exploring literature on approaches for these areas.
  • Duplicates: The system initially produced duplicate quantity labels, which were later removed by the organizers for official ranking. Future work should focus on robust duplicate handling.
  • Math Equalities and Calculations: The custom tokenizer and system struggled with complex mathematical expressions and calculations, leading to errors.
  • Units: The unit gazetteer, while extensive, did not cover all units present in the gold standard, leading to false negatives.
  • Future Directions: The authors suggest further investigation into domain-inspired contributions to support the DGCNN for all task components effectively, and more robust handling of duplicates.

Summary updated

BibTeX
@inproceedings{lathiff2021clac,
  author = {Nihatha Lathiff and Pavel PK Khloponin and Sabine Bergler},
  title = {{CLaC-np at SemEval-2021 Task 8: Dependency DGCNN}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure