In the world of natural language processing, there is a constant tension between expressive power and computational efficiency. Standard Context-Free Grammars (CFGs) are easy to parse in $O(n^3)$ time, but they struggle with the "non-local" dependencies found in translation or complex syntax. To capture these, researchers use more powerful formalisms like Linear Context-Free Rewriting Systems (LCFRS).
However, moving to these "mildly context-sensitive" grammars introduces a significant hurdle: the complexity of the parsing algorithm depends heavily on how the grammar rules are structured. The core strategy to manage this complexity is binarization—the process of breaking down complex rules into a sequence of simpler, binary steps.
The Power of the Bilinear Form
The most efficient parsing algorithms, such as the CKY algorithm for CFGs, rely on rules having at most two non-terminals on the right-hand side. René Leermakers demonstrated in How to Cover a Grammar that many disparate parsing strategies—including Earley and Tomita parsers—can be unified under a single framework if the grammar is transformed into a bilinear cover.
By ensuring every rule is "binary" (at most two non-terminals), we create a predictable structure that dynamic programming can exploit. This is not just a theoretical convenience; it is a practical necessity for tasks like joint parsing and translation. Yang Liu and colleagues showed in Joint Parsing and Translation that decomposing complex tree-to-string rules into binary units allows decoders to explore a massive search space while remaining computationally tractable.
The Fundamental Trade-off: Rank vs. Fan-out
If binarization is so helpful, why not always do it? The problem is that for expressive grammars like LCFRS, binarization isn't free. It introduces a fundamental trade-off between two metrics:
- Rank: The number of non-terminals on the right-hand side of a rule. We want this to be 2.
- Fan-out: The number of disconnected string segments a single non-terminal must track.
When you take a complex rule and "factor" it into binary pieces, you often force the intermediate symbols to keep track of more disjoint pieces of the sentence. I. Dan Melamed, Giorgio Satta, and Benjamin Wellington noted in their work on Generalized Multitext Grammars that reducing the rank of a rule often necessitates an increase in fan-out.
Imagine trying to describe a translation where words from the beginning, middle, and end of a source sentence all cluster together in the target. A single rule could describe this, but to break that rule into binary steps, the intermediate "helper" rules must remember all those disconnected gaps. This increased fan-out directly inflates the state space of the parser.
Why Optimal Binarization is Hard
Finding the "perfect" balance between rank and fan-out is not just difficult—it is often computationally intractable. Giorgio Satta proved as early as 1992 in Recognition of Linear Context-Free Rewriting Systems that recognizing these languages is NP-hard unless we strictly limit both fan-out and production length.
Later, Pierluigi Crescenzi, Daniel Gildea, and their collaborators showed in Optimal Head-Driven Parsing Complexity for Linear Context-Free Rewriting Systems that even finding the most efficient way to parse a single rule is a "hard" problem. Because minimizing the number of symbols (rank) can lead to an exponential increase in the number of segments (fan-out), there is no simple universal algorithm to binarize every grammar optimally.
Practical Paths Forward
Despite these theoretical barriers, researchers use binarization to make high-order parsing possible in real-world applications:
- Nested Named Entity Recognition: Christopher D. Manning and Jenny Rose Finkel treated NER as a parsing task in Nested Named Entity Recognition, using right-branching binarization and Markovization to keep the search space manageable for long sentences.
- Matrix Multiplication: Shay B. Cohen and Daniel Gildea demonstrated in Parsing Linear Context-Free Rewriting Systems with Fast Matrix Multiplication that once a grammar is binarized, we can apply generalizations of Valiant’s algorithm to achieve asymptotic speedups for complex systems like Tree-Adjoining Grammars (TAG).
The research direction across these papers suggests that binarization is the "bridge" that allows us to use sophisticated linguistic models. While the "perfect" binarization remains an open challenge, the act of factoring complex rules into binary operations is what transforms theoretical formalisms into functioning parsers.