A generative extractor reads a sentence and writes out its facts as text, for example "attack event, attacker = X, target = Y". When such a model is trained on English and tested on Chinese or Arabic, it has to write strings in a language its decoder never produced during training. We propose to remove that requirement. Entity mentions in the input are replaced with numbered type placeholders, a technique called entity masking. The decoder is then built to output only positions: which masked slot fills which role, and which input word is the event trigger. It never spells any target-language word. Because these position outputs mean the same thing in every language, everything the decoder learns from English annotations carries over unchanged. All language-specific variation is pushed into the multilingual encoder, which is the part known to carry shared syntax across languages.
The mechanism that makes this work is a pointer-generator, a decoder that at each step either emits a token from a fixed vocabulary or "points" to an input position by attention. Pointer-generators have a poor record in cross-lingual settings. In low-resource translation they copy seemingly random subwords and give at best marginal gains (Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!). Our bet is that this failure comes from what is being copied, namely surface subwords from mismatched tokenizations, and not from pointing as such. When the targets are masked argument slots and trigger positions, and the pointer's attention is biased by distance in the dependency tree (the parse that links each word to its grammatical head), pointing should become the right inductive bias for transfer.
Core Claim
Zero-shot cross-lingual generative event and relation extraction fails mainly because of the output space, not the encoder. If entity masking is combined with a decoder whose role outputs are pointers to masked slots and trigger positions, the decoder becomes language-independent. That should give larger gains on typologically distant targets (Arabic, Chinese, Indic languages) than on close ones. It should also give larger gains on sentences with many arguments. On these sentences, flat slot copying is known to degrade (Delexicalized Paraphrase Generation), and a dependency-distance bias on the pointer should resolve which slot fills which role.
This differs from the closest work in specific ways. Delexicalized Paraphrase Generation copies slot placeholders with a pointer, but only in English, with no syntax, and it reports the multi-slot degradation we target. GATE injects dependency distances into encoder self-attention and uses entity-type tags. However, it is a classifier over given argument candidates and contains no decoder. Our proposal moves the distance prior into the copy distribution. The translation study above points over raw subwords, and its authors suggest that structural constraints are needed. We test that suggestion directly, in an extraction setting where the copy targets are well defined. Multi-Source Transfer of Delexicalized Dependency Parsers delexicalizes every word. Following the finding of Cross-lingual Structure Transfer for Relation and Event Extraction that symbolic features need distributional ones alongside them, we mask only the arguments and keep predicates lexical.
Research Program
The model is a pretrained multilingual encoder-decoder. In its input, each detected entity mention is replaced by a typed, numbered placeholder such as [PER_2]. The surface tokens are kept in a side table, so that the final output can be mapped back to the original mentions. The output is a linearized event record. Its fixed-vocabulary tokens are only schema symbols: event types, role names, and separators. Its argument and trigger tokens are pointer actions over input positions.
At the step where the decoder fills a role for an event whose trigger sits at position $\tau$, the copy distribution over input position $i$ is
$$a_i \propto \exp\big(e_i - \lambda\, D_{i\tau}\big),$$
where $e_i$ is the ordinary attention score, $D_{i\tau}$ is the shortest-path length between token $i$ and the trigger in a Universal Dependencies parse, and $\lambda \ge 0$ is learned. The gate that chooses between pointing and generating is the standard one from Get To The Point, with one change: schema tokens can only be generated and argument tokens can only be pointed. This makes the gate a structural switch and not a learned guess, which sidesteps the "extractive inertia" that paper documents.
Parses come from an off-the-shelf multilingual parser. Parser error in low-resource languages is a known weakness of syntax-based transfer (Fei et al.). To keep this from silently driving the results, $\lambda$ is learnable, so the model can down-weight the prior. We also test with deliberately corrupted parses.
Masking requires mentions. For the end-to-end setting we therefore train a multilingual mention detector on English and run it on the target language. Its errors propagate into the extractor, and we report them separately.
Baselines
The naive reference point is a multilingual encoder-decoder fine-tuned on English to generate linearized events in surface form, with no masking and no pointer, then applied directly to Chinese and Arabic. This is what most people would try first. We report its raw F1 and also a "script leakage" rate, meaning the fraction of predicted arguments that are not substrings of the source sentence. Leakage measures the failure we hypothesize.
The strongest existing methods on this benchmark are discriminative syntax-aware encoders evaluated on ACE 2005 with gold argument candidates. GATE reports 68.5% F1 on English-to-Arabic event argument role labeling, against 50.4% for CL-GCN. Cross-lingual Structure Transfer reports 59.0% F1 English-to-Chinese, close to its 59.3% target-supervised result. For relation extraction, Fei et al. report 67.2% average F1 (63.7% Chinese, 70.7% Arabic). We reimplement GATE on our encoder with the same parses and the same tuning budget. Matching its gold-candidate number shows that our model gives up nothing by being generative. Beating it is not required in that setting. The claim to beat is in the end-to-end setting: we feed GATE the same predicted mentions and triggers as our model, and our model should exceed it there by a margin larger than seed variance.
The key ablation keeps entity masking but replaces pointing with ordinary generation of the placeholder tokens ([PER_2] produced from the vocabulary). Triggers are produced as surface words. A second ablation removes only the dependency bias by setting $\lambda = 0$. All systems share one backbone, the English training split, early stopping on English development data only, and an identical hyperparameter search budget.
Why not just…?
Why not just fine-tune a multilingual seq2seq model to generate surface tuples? Chi et al. show that multilingual encoders already carry shared syntactic subspaces, so the encoder side may be fine. But nothing in that result says a decoder trained only on English outputs can spell Arabic arguments faithfully. The naive baseline plus the leakage metric settles this. If leakage is low and F1 matches ours, the output-space hypothesis is wrong.
Why not just constrain decoding to source spans? Constrained decoding guarantees faithful outputs without any architectural change, and it is a real threat. It still forces the decoder to emit target-language subword sequences whose output embeddings were never trained. Pointing replaces that with a single index decision. We run constrained decoding as a full baseline. If it matches the pointer model, we will report that the pointer is unnecessary and that masking plus a fidelity constraint is sufficient.
Why not just mask and generate the numbered placeholders, with no pointer at all? This is the strongest threat, because numbered placeholders are already language-agnostic. The pointer can still help in two places. One is triggers, which stay lexical and so cannot be placeholders. The other is sentences with many arguments, where Shikder et al. saw flat slot reproduction collapse. If the masked-generation ablation matches the full model even on the subset with four or more arguments, we will drop the pointer from the claim.
Why not just prompt a large language model? Dutt et al. found zero-shot LLMs significantly worse than fine-tuned cross-lingual models on IndoRE and REDFM, with verbalized dependency paths adding only 1.67% F1. We include one prompted reference for context but do not treat it as the main competitor.
Proposed Experiments
The main benchmark is ACE 2005, training on English and testing zero-shot on Chinese and Arabic. We evaluate argument role labeling and relation extraction, each in two settings: gold-candidate and end-to-end. For a lower-resource, wider-typology check we use IndoRE (Bengali, Hindi, Telugu) and REDFM (seven languages). Here we only have relative improvements from Dutt et al. and no absolute F1 from the file, so all baselines are rerun ourselves. Metrics are role F1, tuple F1, script leakage rate, and F1 stratified by number of arguments per sentence and by dependency distance between trigger and argument.
Three outcomes decide the claim. First, the gap between pointer-over-masked-slots and surface generation should be larger for Arabic and Chinese than for typologically closer REDFM languages such as Spanish and Italian. If the gap is flat across languages, the output-space story fails. Second, the dependency bias should help most on many-argument and long-distance subsets. With shuffled parses, that benefit should disappear and $\lambda$ should shrink toward zero. If shuffled parses help equally, the bias is acting as a regularizer and not as syntax. Third, in the end-to-end setting, our model should beat GATE given identical predicted mentions. We will also measure how much of any remaining gap is due to the mention detector, by swapping in gold mentions one component at a time.
References
- Pointer-Generator Networks for Low-Resource Machine Translation: Don’t Copy That!
- Delexicalized Paraphrase Generation
- GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction
- Multi-Source Transfer of Delexicalized Dependency Parsers
- Cross-lingual Structure Transfer for Relation and Event Extraction
- Get To The Point: Summarization with Pointer-Generator Networks
- Constructing Code-mixed Universal Dependency Forest for Unbiased Cross-lingual Relation Extraction
- Finding Universal Grammatical Relations in Multilingual BERT
- Can dependency parses facilitate generalization in language models? A case study of cross-lingual relation extraction