Post-hoc Manipulations of Vector Space Models with Application to Semantic Role Labeling

Jenna Kanerva, Filip GinterCvsc@eacl 2014 · DOI

Introduction

Semantic Role Labeling (SRL) is a fundamental task in natural language processing that aims to identify the underlying predicate-argument structure of a sentence. Essentially, it answers the question: "Who did what to whom, where, and when?" While modern deep learning has pushed the boundaries of SRL, these systems often rely on complex, task-specific architectures and extensive feature engineering.

This research explores a more minimalist approach: can we "hack" general-purpose, pre-trained word vectors (like those from word2vec) to perform SRL using simple algebraic operations? By applying post-hoc manipulations such as circular shifts and linear transformations, the authors demonstrate that syntactic and semantic structure can be effectively injected into vector space models (VSMs) without retraining the underlying embeddings.

The Challenge of Semantic Role Labeling

In a standard SRL task, a system must identify a predicate (the action) and its associated arguments (the participants). For example, in the sentence "He ate lunch," "ate" is the predicate, "He" is the agent (A0), and "lunch" is the patient or object (A1).

Figure 1: An example of a dependency tree annotated with semantic roles (A0, A1, AM-TMP).

As shown in Figure 1, these roles are often tied to syntactic dependency relations. However, standard word vectors are typically "bag-of-words" representations; they capture lexical meaning but are blind to the specific grammatical role a word plays in a given sentence. To solve SRL, a model must address two primary hurdles:

  1. Encoding Syntax: Distinguishing between a word acting as a subject versus an object.
  2. Cross-Linguistic Robustness: Ensuring the method works for morphologically rich languages like Finnish, where word forms are significantly more complex than in English.

Methodology: Post-hoc Vector Manipulations

The researchers proposed three primary techniques to manipulate existing word vectors after their initial training phase:

1. Circular Shifting for Syntax

To encode structural information, the authors used a computationally trivial operation: the circular shift. If a word is identified by a dependency parser as a subject, its vector dimensions are shifted by a fixed offset. If it is an object, it is shifted by a different offset. This allows the same word vector to occupy different regions of the high-dimensional space depending on its grammatical context, effectively "tagging" the vector with its syntactic role.

2. Average Argument Vectors (Prototypes)

To represent the "typical" participant of an action, the authors created prototype vectors. For a specific verb like eat, they averaged the vectors of all words found in the object position (e.g., apple, soup, bread) within a training corpus. To classify a new word, the system simply calculates the cosine similarity between the candidate word's vector and these role-specific prototypes.

3. Linear Transformations

Building on the linear properties of VSMs (e.g., the famous "king - man + woman = queen" analogy), the authors learned transformation matrices ($W$). These matrices map a predicate vector directly to the expected vector space of its arguments ($W \cdot \text{predicate} \approx \text{argument}$), allowing the model to "predict" the semantic profile of a missing participant.

Results and Analysis

The experiments were conducted on the English CoNLL’09 dataset and the Finnish PropBank. For Finnish, the authors introduced a new 1.5-billion-token corpus derived from Common Crawl to ensure robust vector representations.

Figure 5: Ablation study showing the performance impact of removing syntax (circular shifts) and other features across different methods.

As detailed in Figure 5, the inclusion of syntactic information via circular shifts provided a significant performance boost. For the "Average vectors" method in Finnish, the F-score dropped from 66.23% to 61.32% when syntactic information was removed. This confirms that circular shifting is a remarkably effective way to inject structural awareness into lexical vectors.

When these vector-based similarity scores were integrated as features into a supervised SVM classifier, the system achieved an F-score of 73.83% for Finnish and 70.38% for English. In a full system pipeline, the method reached an F-score of 82.33% on English, coming within 4 percentage points of state-of-the-art systems that rely on much more complex feature sets.

Conclusion and Future Work

This study demonstrates that the linear and structural properties of vector space models are more powerful than previously thought. Simple post-hoc manipulations can bridge the gap between lexical semantics and functional syntax. While the "one-to-many" problem—where a single verb like see can take highly diverse arguments—remains a challenge for single-prototype models, the success of these algebraic hacks suggests a promising path for lightweight, cross-linguistic semantic parsing. Future work may involve using clustering to represent multiple argument prototypes for a single predicate, further refining the model's ability to capture the diversity of natural language.

Summary updated

BibTeX
@inproceedings{kanerva2014post,
  author = {Jenna Kanerva and Filip Ginter},
  title = {{Post-hoc Manipulations of Vector Space Models with Application to Semantic Role Labeling}},
  booktitle = {Cvsc@eacl},
  year = {2014},
  doi = {10.3115/v1/w14-1501},
}
Figure