Bayesian Phrase Posteriors for Rare Multi-Word Expressions

Bilingual phrase dictionaries are still useful in the neural era. Statistical phrase tables mined from word alignments translate isolated idioms and collocations far better than sentence-level neural translators: 69.2% versus 38.4% accuracy on a Russian-English benchmark of 250 expressions (Khusainova et al.). Those tables are built with a brittle two-step recipe. First a word aligner links individual words across each sentence pair. Then a fixed rule, "heuristic phrase extraction," keeps every pair of spans whose internal word links do not leak outside the box. Counts are filtered by frequency thresholds. The same paper reports that the recipe degrades badly once an expression occurs fewer than about ten times. That is where most idioms and domain collocations live. At such low counts, one wrong word link inside a non-compositional phrase removes the correct pair entirely, and there is too little repetition left to average the error away.

A different line of work, joint alignment and extraction, avoids the word-alignment step altogether. It uses Bayesian synchronous grammars that segment and align both sentences at once and can store a whole idiom as a single unit (Neubig et al.; Cohn and Haffari). These models were built to produce compact translation tables for full statistical translation systems. They were never pointed at dictionary building, and they are far too slow to run over a 140-million-sentence corpus. We propose using each family for what it does well. Cheap heuristic extraction decides where to look. A joint Bayesian model, run only on the handful of sentence pairs containing a candidate expression, decides what the expression translates to. Its averaged evidence about whole-span correspondences replaces the frozen word links that fail at low counts.

Core Claim

For bilingual expressions with fewer than roughly ten corpus occurrences, a targeted joint Bayesian phrase model gives more accurate dictionary entries than heuristic extraction from word alignments. The model is restricted to sentence windows proposed by the heuristic pipeline, and it scores each candidate translation by its posterior probability of being generated as one atomic unit. The gain should come specifically from scoring whole spans jointly rather than from better word links. For frequent expressions, the two approaches should converge, because repetition already washes out alignment noise.

The design differs from its closest relatives in three ways. Khusainova et al. keep word alignments frozen and only filter counts afterward; we rescore candidates with a model that never commits to word links. Neubig et al. and Cohn and Haffari induce entire phrase tables for decoding and evaluate by BLEU; we run inference per expression on small subcorpora and evaluate dictionary accuracy stratified by frequency. That is the regime their caching prior should help most, and it had not been measured. Shen et al. push word alignments toward phrase-consistent shapes, but extraction stays heuristic. Here heuristic extraction is demoted to a search-space proposal.

Research Program

The pipeline has three stages.

The first stage, proposal, runs the standard pipeline at scale: word alignment followed by grow-diag-final-and extraction. For each source expression it collects every sentence pair containing the expression, together with the target spans that heuristic extraction associates with it. We deliberately loosen this step to favour recall. We take the union of the two directional alignments and also add target spans that are consistent under alignments with single links removed. This addresses the failure documented by Deng et al., where one bad link blocks a valid pair.

The second stage, local joint inference, crops each occurrence to a window of about ±10 tokens around the expression and its candidate target region. The cropping keeps the cost of the chart that pairs every source span with every target span (cubic in each sentence's length) manageable. The model is a hierarchical Pitman-Yor inversion transduction grammar of the HIER type. In plainer terms, it is a grammar that builds both sentences by recursively splitting them into aligned pieces, kept in or swapped in order. A "rich-get-richer" prior stores frequently reused bilingual pieces as whole units and builds rare ones from smaller parts (Neubig et al.). Cohn and Haffari used external word alignments to prune the chart; we prune with the loosened candidate set from stage one.

A crucial detail is that the windows for one expression are sampled jointly, sharing one cache. Each further occurrence then raises the probability that the same atomic pair explains the others. Base distributions are trained once on a moderate background subcorpus, so rare expressions fall back to sensible word-level translation probabilities. We follow Cohn and Haffari in averaging over several sampling chains rather than taking one best derivation, because their results show that chains capture alignment modes a single derivation misses.

The third stage, scoring, ranks candidate target spans for each expression. The score is the average posterior probability, over samples and occurrences, that the source span and that target span appear as one node in the derivation. Inverse-direction scores are combined as Khusainova et al. combine their two translation probabilities.

Beyond the headline comparison, the program analyses why any gain appears. For each expression we record whether the heuristic baseline's error came from the correct pair being blocked (no consistent box exists), from it being outranked, or from it never co-occurring. We then measure which of these error types the joint model repairs.

Baselines

The naive reference point is the sentence-level neural translator applied to the isolated expression. Khusainova et al. report 38.4% any-match accuracy for MarianMT on their 250-expression Russian-English benchmark. We rerun it on our frequency-stratified sets to get per-bucket numbers.

The strongest existing method is the Khusainova et al. pipeline itself: GIZA++ alignment of Russian-English CCMatrix v1, heuristic extraction, and dynamic frequency-dependent thresholds over counts and both translation directions. It reports 69.2% any-match and 62.4% top-1 accuracy on the 250-expression benchmark. We rerun it on exactly the same corpus slice and test sets, retuning its thresholds on the same development split we use for our own few hyperparameters (window size, prior strength).

Beating it means a statistically significant top-1 improvement on the rare bucket (counts below 10), measured by paired bootstrap over expressions. It also means no significant loss on the frequent bucket and no loss on the original 250-item benchmark. A gain confined to the rare bucket is the predicted outcome. A uniform gain would suggest that something other than the claimed mechanism is at work, and we would investigate it.

Three ablations isolate the key ingredient. The first keeps our loosened candidate set but scores candidates with heuristic relative frequencies. This tests whether recall in the proposal stage alone explains the gain. The second keeps the joint model but samples each occurrence independently, without a shared cache. This tests whether pooling evidence across occurrences matters. The third uses one chain's single best derivation instead of chain-averaged posteriors. All variants receive the same candidate sets and the same development tuning budget. The joint model's compute is reported per expression so its cost is visible.

Why not just…?

Why not just swap in a better neural word aligner and keep heuristic extraction? This is the most serious threat. Attention-based alignment induction such as SHIFT-AET reports AER 1.4 to 4.8 points below GIZA++ (Chen et al.), and hidden-state prediction difference also beats statistical aligners (Li et al.). If fewer bad links are all that rare expressions need, this alternative would match us at lower cost. Two pieces of evidence argue against assuming so. Lower AER does not reliably give better phrases (Vilar et al.). And non-compositional idioms have no correct word-level links to recover in the first place. We found no reported result on Lacuna for neural aligners on MWE dictionary extraction, so we run SHIFT-AET plus the Khusainova filtering as a full baseline. We also run our joint scorer on top of SHIFT-AET candidates. If the neural-aligner pipeline closes the rare-bucket gap, we will report that joint modelling is unnecessary for this task.

Why not just regularize the word alignment toward phrase-consistent shapes? Consistency-aware search gives about 1.0 BLEU on Chinese-English (Shen et al.), and it would directly reduce blocked pairs. It still scores phrases by counted frequency, though, so it offers nothing new where there are fewer than ten counts to rank with. We include it as a proposal-stage variant. Our error analysis shows whether it repairs "blocked" errors while leaving "outranked" ones, which is what we predict.

Why not just mine more data or lower the thresholds? CCMatrix is already about 140M sentences. Phrase frequencies are heavy-tailed, so more data shifts the rare bucket rather than emptying it. Lowering thresholds trades precision for recall, and Khusainova et al. already tuned this trade-off. The threshold sweep in our tuned baseline settles the question empirically.

Why not just run the full joint model on the whole corpus? Cohn and Haffari had to cap sentences at 30 words even with pruning, so corpus-scale inference is impractical. The targeted design is the proposal's answer to that cost. The "no shared cache" ablation tests whether local inference loses the benefit of global pooling.

Proposed Experiments

The main testbed is Russian-English CCMatrix v1 with the 250-expression benchmark of Khusainova et al. Because that set is small and probably skewed toward frequent items, we will build a frequency-stratified extension. Bilingual annotators will judge translations for about 600 additional Russian idioms and collocations sampled to fill three count buckets (under 10, 10–100, over 100). Annotation will be blind to which system produced each candidate, and we report any-match and top-1 accuracy per bucket.

A second, intrinsic check uses the hand-aligned NIST Chinese-English data of DeNero and Klein. We compute Bispan $F_5$, a phrase-pair F-measure weighted toward recall, for our posterior-thresholded extraction. GIZA++ scores 46.0 there and their supervised extraction-set model 74.0. We use their number only as a supervised ceiling, since our method sees no gold links.

The decisive checks are threefold. The rare-bucket top-1 gain over the retuned Khusainova pipeline and over the SHIFT-AET pipeline must survive paired bootstrap. The shared-cache ablation must lose most of that gain. And the error analysis should show recovered entries concentrated among non-compositional expressions. If the neural-aligner baseline matches us on the rare bucket, the core claim fails, and the paper becomes a negative result about when joint models still matter.