Published in the late 1980s, this paper explores an early connectionist (neural network) approach to a classic headache in linguistics: translating locative prepositions. By utilizing a shared semantic "interlingua," the authors demonstrate how a simple backpropagation network can capture the nuanced, context-dependent meanings of spatial words like "on," "at," and "in" across English and German.
The Challenge of Prepositions
Prepositions are notoriously difficult to translate because they rarely share a one-to-one mapping between languages. For example, the English word "on" might translate to the German auf (indicating placement on top of a horizontal surface) or an (indicating attachment to a vertical surface or edge).
Traditional machine translation systems often struggled with this ambiguity because the "correct" preposition depends heavily on the surrounding nouns—the context. This paper investigates whether a connectionist model, a precursor to modern deep learning, can learn these subtle spatial relationships and use a non-linguistic "spatial representation" as a bridge between languages.
Methods: The "Interlingua" Strategy
The researchers designed a multi-layer neural network trained via backpropagation to map "noun-preposition-noun" phrases to a shared spatial semantic representation.
- Architecture: The network takes a linguistic triple as input:
[Noun 1 - Locative Preposition - Noun 2]. It also includes a dedicated Spatial Relationship (SR) component. - Encoder Banks: To process the nouns, the authors used "encoder banks" (inspired by Geoffrey Hinton’s early work). These forced the network to develop its own internal, distributed representations of the nouns rather than treating them as isolated, meaningless labels.
- Two-Step Translation: Translation was treated as a decoding-encoding process bridged by an "interlingua."
- Step 1 (Decoding): An English-trained network receives a phrase (e.g., "house on lake") and generates a semantic Spatial Representation (SR).
- Step 2 (Encoding): This SR, along with the translated nouns, is fed into a German-trained network, which then predicts the most appropriate German preposition (in this case, an).
To train the networks, the authors used a "pattern-completion" task. The models had to learn to fill in whichever piece of the puzzle was missing, whether that was the preposition itself or the underlying spatial meaning.
Datasets and Spatial Features
To test their hypothesis, the authors constructed a constrained "toy" universe.
- Vocabulary: The dataset consisted of 25 nouns (e.g., clouds, lake, table, fish) and 5 prepositions per language.
- Prepositions: The English set included in, at, on, under, above, while the German set included in, an, auf, unter, über.
- Training Set: Out of 3,125 possible combinations, the researchers selected 137 "meaningful" expressions to form the training corpus.
- Spatial Features: Drawing on linguistic theories by Annette Herskovits, each phrase was mapped to a set of 10 spatial relationship units (e.g., "N1 touching N2," "N2 supports N1").
Key Insights
- The "Ideal Meaning": By presenting a preposition to the network without any accompanying nouns, the authors could visualize the network's "internal prototype" for that word. They discovered that the German auf was essentially treated as a hybrid of the English on and at, lacking the specific "edge" or "nearness" components found in an.
- Nouns as Modifiers: The model successfully captured how nouns constrain prepositional meaning. The network didn't just memorize that "on" equals auf; it learned that the specific geometry of a "house" and a "lake" necessitated the use of an.
- Asymmetric Learning: Interestingly, the English networks learned the training corpus more accurately than the German ones. This suggests that the mapping between syntax and spatial semantics may be more "regular" or consistent in English for this specific set of locative phrases.
Results and Limitations
The translation performance was remarkably high for such an early connectionist model. For English to German translation, the system averaged only 1.75 errors out of 137 phrases (approximately 98.7% accuracy). The German to English direction proved slightly more difficult, averaging 6.75 errors (approximately 95% accuracy). All networks converged to very low error states, proving that backpropagation could effectively capture the "interlingua" mapping.
Despite these successes, the study had notable limitations. The dataset was very small, and the authors acknowledged that representing "arbitrary semantics" for a full-scale language is a vastly larger challenge. Furthermore, the study assumed that nouns are context-independent and easy to translate, which is rarely true in practice. Finally, the authors noted that future NLP models would need to process temporal sequences of words rather than static triples—a prediction that foreshadowed the rise of Recurrent Neural Networks (RNNs) in the following decade.
Summary updated