Abstract
A fundamental problem in Natural Language Processing (NLP) is the integration of syntactic and semantic constraints. This paper describes a new approach for this integration which takes advantage of a learned memory model. The model combines localist representations for the integration of constraints and distributed representations for learning semantic constraints. Applied to structural disambiguation of noun phrases, this connectionist model demonstrates an ability to scale up the underlying memory of NLP systems.
The Challenge of Structural Ambiguity
In NLP, structural ambiguity is a classic hurdle, particularly the "prepositional phrase (PP) attachment" problem. This task asks: what does a prepositional phrase actually modify? Consider the complex noun phrase: "The influence of the radiation intensity on discharges in the Van-Allen-belt." Does "in the Van-Allen-belt" modify "discharges," "intensity," or "influence"?
While humans navigate this using a mix of semantic intuition and syntactic rules, machines often struggle. Most traditional systems focused on single PPs within full sentences. This research tackles the more difficult task of disambiguating multiple nested PPs within long noun phrases, common in scientific literature where verbal context is often missing.
A Two-Level Connectionist Architecture
The author proposes a hybrid connectionist architecture that operates on two distinct levels to resolve these ambiguities.
1. Semantic Memory (Bottom Level)
The bottom level utilizes standard backpropagation networks with distributed representations. For each preposition (such as "in," "of," or "on"), a dedicated network is trained to recognize if a relationship between two nouns is "plausible" or "implausible."
Nouns are represented as 16-dimensional binary vectors based on semantic features like ENERGY, TIME, PHYSICAL-OBJECT, and CHANGING-EVENT. The network learns a "memory" of which semantic categories typically pair well with specific prepositions.
2. Syntactic Integration (Top Level)
The top level is a relaxation network using localist representations, where each node represents a specific concept or constraint. It integrates three types of information to reach a stable structural interpretation:
- Semantic Plausibility: Fed directly from the bottom-level networks.
- Locality Constraint: A heuristic suggesting a PP is more likely to attach to a nearby noun than a distant one.
- No-crossing Constraint: A structural rule stating that attachment branches should not cross each other.
Key Results and Insights
The system was evaluated using the NPL corpus, which consists of article titles from physical science domains. Key findings include:
- Semantic Accuracy: The backpropagation network for the preposition "in" achieved 93% accuracy on its training set and 83% accuracy on unknown testing examples.
- Semantics Overriding Syntax: A critical finding is that strong semantic evidence (e.g., a pairing being highly implausible) can successfully override syntactic preferences like the locality heuristic.
- Structural Disambiguation: In the complex "Van-Allen-belt" example, the relaxation network correctly identified the preferred structural hierarchy by balancing the implausibility of certain pairings against syntactic rules.
- Generalization: Because the system relies on semantic features rather than specific words, it can disambiguate noun phrases it has never seen before, provided the nouns fit into the known 16-feature taxonomy.
Conclusion
This hybrid approach demonstrates that combining distributed and localist neural architectures provides a robust framework for structural disambiguation. By using neural networks to learn semantic constraints rather than hard-coding thousands of rules, the system becomes more scalable and better at handling the "fuzzy" data inherent in natural language.
Summary updated