Introduction
Governments worldwide are legally mandated to release official documents under Freedom of Information (FOI) laws. However, a significant manual bottleneck exists: "sensitivity review." Documents must be meticulously checked to protect personal privacy or national security interests before public release. This manual process is time-consuming and prone to inconsistency.
The primary challenge for automated sensitivity classification is that sensitivity is inherently context-dependent. For instance, a senior official's salary might be public record, whereas a junior employee's salary is strictly private. Standard Knowledge Graph Embedding (KGE) techniques typically learn separate vectors for entities and relations. Because these vectors are generalized across an entire graph, they often fail to capture the specific nuances of individual relationship triples. To address this, we introduce RelDiff (Relationship Difference), a novel approach that creates unified "triple embeddings" (head-relation-tail) to better capture context for sensitivity classification.
Methodology
The RelDiff framework transforms standard KGEs into descriptive features for classification through a multi-stage process involving extraction, composition, and ensemble classification.
Relation Extraction and Vector Composition
Initially, a reinforcement learning-based model (HRL-RE) extracts entity-relation triples from the document text. RelDiff then composes a single embedding for each triple $(h, r, t)$ using two primary algebraic operators:
- Hadamard Product ($\odot$): The head ($v_h$) and tail ($v_t$) entity vectors are multiplied element-wise. This operation filters and scales shared features, resulting in a semantic "pair vector."
- Vector Subtraction: This pair vector is projected into the relation's subspace and subtracted from the relation vector ($v_r$). This captures the specific "direction" and distance between a particular entity pair and the generalized relation type.
Classification Pipeline
The system employs a "stacking ensemble" architecture. One classifier processes traditional text features (Bag-of-Words), while another analyzes the RelDiff embeddings. A meta-classifier then integrates these outputs to produce the final sensitivity determination.
Experimental Setup
The primary benchmark for this research is the GovSensitivity dataset, comprising 3,801 UK government documents manually reviewed for FOI sensitivities, specifically focusing on Personal Information and International Relations. To enhance model robustness, the authors utilized FB15k-237 (a Freebase subset) for pre-training KGE models before fine-tuning on the document-specific graph. The initial relation extraction model was trained on the NYT10 dataset.
Results and Discussion
RelDiff consistently outperformed standard KGE methods, including TransE, RotatE, and HAKE, across all experimental configurations.
Performance Metrics
The RelDiff-RotatE configuration achieved the highest performance, with an F1 score of 0.426 and a Balanced Accuracy (BAC) of 0.739. This represents a significant improvement over the text-only baseline, which recorded 0.409 F1 and 0.728 BAC.
Practical Impact
Beyond statistical metrics, the practical utility of RelDiff is substantial. In a simulated review environment, the RelDiff-RotatE model converted 77 False Positives into True Negatives compared to the baseline. Given standard human review speeds, this translates to a saving of approximately 2.5 hours for a small batch of 3,800 documents. Scaled to the millions of documents handled by government agencies, the potential for time and cost savings is immense.
Limitations
The system's efficacy is currently bounded by the quality of the initial Relation Extraction; failures in identifying relationships leave RelDiff with no data to process. Furthermore, performance gains varied by relation type; while relations like place_of_birth saw improvements, others like founder/organisation experienced slight degradation, likely due to varying relation frequencies in the training data.
Conclusion
RelDiff demonstrates that for sensitivity classification, the "triple is the message." By using simple yet effective algebraic operators to fuse entity and relation embeddings, RelDiff captures the contextual nuances necessary for accurate sensitivity review. Future work will focus on improving relation extraction robustness and investigating the differential impact of RelDiff across various relation types.
Summary updated