RelDiff: Enriching Knowledge Graph Relation Representations for Sensitivity Classification

Hitarth Narvala, Graham McDonald, Iadh OunisEMNLP 2021 · DOI

Introduction

Governments worldwide are legally mandated to release official documents under Freedom of Information (FOI) laws. However, a significant manual bottleneck exists: "sensitivity review." Documents must be meticulously checked to protect personal privacy or national security interests before public release. This manual process is time-consuming and prone to inconsistency.

The primary challenge for automated sensitivity classification is that sensitivity is inherently context-dependent. For instance, a senior official's salary might be public record, whereas a junior employee's salary is strictly private. Standard Knowledge Graph Embedding (KGE) techniques typically learn separate vectors for entities and relations. Because these vectors are generalized across an entire graph, they often fail to capture the specific nuances of individual relationship triples. To address this, we introduce RelDiff (Relationship Difference), a novel approach that creates unified "triple embeddings" (head-relation-tail) to better capture context for sensitivity classification.

Methodology

The RelDiff framework transforms standard KGEs into descriptive features for classification through a multi-stage process involving extraction, composition, and ensemble classification.

Relation Extraction and Vector Composition

Initially, a reinforcement learning-based model (HRL-RE) extracts entity-relation triples from the document text. RelDiff then composes a single embedding for each triple $(h, r, t)$ using two primary algebraic operators:

  1. Hadamard Product ($\odot$): The head ($v_h$) and tail ($v_t$) entity vectors are multiplied element-wise. This operation filters and scales shared features, resulting in a semantic "pair vector."
  2. Vector Subtraction: This pair vector is projected into the relation's subspace and subtracted from the relation vector ($v_r$). This captures the specific "direction" and distance between a particular entity pair and the generalized relation type.
Geometric representation of the RelDiff embedding process, showing the interaction between entity and relation subspaces.

Classification Pipeline

The system employs a "stacking ensemble" architecture. One classifier processes traditional text features (Bag-of-Words), while another analyzes the RelDiff embeddings. A meta-classifier then integrates these outputs to produce the final sensitivity determination.

Experimental Setup

The primary benchmark for this research is the GovSensitivity dataset, comprising 3,801 UK government documents manually reviewed for FOI sensitivities, specifically focusing on Personal Information and International Relations. To enhance model robustness, the authors utilized FB15k-237 (a Freebase subset) for pre-training KGE models before fine-tuning on the document-specific graph. The initial relation extraction model was trained on the NYT10 dataset.

Results and Discussion

RelDiff consistently outperformed standard KGE methods, including TransE, RotatE, and HAKE, across all experimental configurations.

Performance Metrics

The RelDiff-RotatE configuration achieved the highest performance, with an F1 score of 0.426 and a Balanced Accuracy (BAC) of 0.739. This represents a significant improvement over the text-only baseline, which recorded 0.409 F1 and 0.728 BAC.

Performance comparison of RelDiff and Concat baselines across varying C parameters for BAC and F1 metrics.

Practical Impact

Beyond statistical metrics, the practical utility of RelDiff is substantial. In a simulated review environment, the RelDiff-RotatE model converted 77 False Positives into True Negatives compared to the baseline. Given standard human review speeds, this translates to a saving of approximately 2.5 hours for a small batch of 3,800 documents. Scaled to the millions of documents handled by government agencies, the potential for time and cost savings is immense.

Limitations

The system's efficacy is currently bounded by the quality of the initial Relation Extraction; failures in identifying relationships leave RelDiff with no data to process. Furthermore, performance gains varied by relation type; while relations like place_of_birth saw improvements, others like founder/organisation experienced slight degradation, likely due to varying relation frequencies in the training data.

Conclusion

RelDiff demonstrates that for sensitivity classification, the "triple is the message." By using simple yet effective algebraic operators to fuse entity and relation embeddings, RelDiff captures the contextual nuances necessary for accurate sensitivity review. Future work will focus on improving relation extraction robustness and investigating the differential impact of RelDiff across various relation types.

Summary updated

BibTeX
@inproceedings{narvala2021reldiff,
  author = {Hitarth Narvala and Graham McDonald and Iadh Ounis},
  title = {{RelDiff: Enriching Knowledge Graph Relation Representations for Sensitivity Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.311},
}
Figure