A Distributed Platform for Sanskrit Processing

Pawan Goyal, Gérard P. Huet, Amba P. Kulkarni, Peter M. Scharf, Ralph BunkerCOLING 2012 · OpenReview

Abstract

Sanskrit presents unique challenges for Natural Language Processing (NLP) due to its complex morphophonetics, where adjacent words merge in writing, and its highly inflected, free-word-order syntax. This paper details a distributed, web-service-based platform that integrates independent Sanskrit processing tools—including segmenters, morphological analyzers, and constraint-based dependency parsers. By unifying these disparate resources, the platform enables the semi-automatic creation of a large-scale tagged corpus, overcoming the limitations of traditional positional NLP models.

Introduction

Most modern NLP tools are optimized for weakly inflected, positional languages like English. Sanskrit, however, violates the fundamental assumptions of these tools. To address this, researchers have developed various specialized digital resources, including lexicons, parsers, and text archives. Historically, these tools remained fragmented across global research groups, utilizing different programming languages and linguistic frameworks. This paper presents a collaborative architecture that uses interoperable web services to unify these tools into a cohesive pipeline for Sanskrit text analysis.

The Computational Challenges of Sanskrit

Sanskrit processing is hindered by three primary linguistic features:

  1. Sandhi (Morphophonetics): Phonetic changes at word junctions are represented explicitly in writing. For example, the sequence vasati + atra becomes vasatyatra. This process obscures word boundaries, making basic tokenization a complex combinatorial problem.
  2. Inflectional Morphology: Sanskrit is exceptionally productive. A single verbal root can yield hundreds of distinct forms. Consequently, a full-form lexicon can exceed 100 million entries, making exhaustive enumeration computationally impractical.
  3. Non-Positional Syntax: Syntactic roles are defined by morphological suffixes rather than word order. Standard constituency parsers, which rely on word position, are largely ineffective for Sanskrit’s free word order.

Methodology and System Architecture

The authors developed a distributed architecture using XML, XHTML, and remote procedure calls (RPC) to connect independent tools without requiring a unified codebase. The system integrates several key methodologies:

Cross-Lingual Lexicon Mapping

To create a comprehensive lexical resource, the team merged the French-based Sanskrit Heritage dictionary with the English-based Monier-Williams dictionary. They utilized automated translation to convert French concepts to English, then treated definitions as "bags-of-words" to align headwords across different ontologies based on semantic similarity.

Segmentation and Sandhi Resolution

To resolve word boundaries, the system employs a functional programming toolkit based on "effective Eilenberg machines" and decorated tries. This allows the engine to efficiently reverse sandhi rules over data streams, identifying valid word segments from merged strings.

Constraint-Based Dependency Parsing

The system utilizes the Samsādhanī parser, which models sentence structure as a directed tree search. It applies ancient Paninian grammatical constraints—specifically ākāṅkṣā (syntactic expectancy) and sannidhi (proximity). These linguistic rules are solved using Minion, a generic constraint solver, to identify the most likely dependency structures.

System Pipelining and Integration

A critical innovation of this platform is the pipelining of a fast, shallow segmenter (the Heritage engine) into a deep, constraint-based parser (Samsādhanī). The Heritage engine acts as a high-efficiency filter, discarding approximately 90% of morphologically invalid sandhi splits before the computationally intensive deep parser begins its analysis. This tiered approach drastically reduces the search space and improves processing speed.

Results and Impact

The integration of the Sanskrit Heritage platform, Samsādhanī, and the Sanskrit Library archives has resulted in a unified, user-friendly web interface for scholars.

One of the most significant outcomes is a machine-assisted annotation tool. When a human scholar validates a single machine-generated segment, the tool automatically prunes all inconsistent solutions from the forest of possible segmentations. This interactive feedback loop allows for the rapid creation of a morphologically and syntactically tagged Sanskrit corpus—a resource that did not exist at scale prior to this collaboration.

Conclusion and Future Work

The platform demonstrates that distributed interoperability and generative grammatical rules are more effective for Sanskrit than the enumerative, positional models used in Western NLP. While the current system is highly efficient, future work will focus on integrating yogyatā (semantic compatibility) constraints to further refine parsing accuracy. The ultimate goal remains the completion of a massive, tagged corpus, which will eventually enable the training of statistical models for higher-level tasks such as anaphora resolution and discourse analysis.

Summary updated

BibTeX
@inproceedings{goyal2012distributed,
  author = {Pawan Goyal and Gérard P. Huet and Amba P. Kulkarni and Peter M. Scharf and Ralph Bunker},
  title = {{A Distributed Platform for Sanskrit Processing}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
  url = {https://openreview.net/forum?id=BtRHfRV6Gf},
}
Figure