Accessible Sanskrit: A Cascading System for Text Analysis and Dictionary Access

Giacomo De LucaNAACL 2025

TL;DR

This paper introduces "Accessible Sanskrit," a novel cascading system for analyzing Sanskrit texts. It tackles the complexity of Sanskrit's morphology and Sandhi (sound changes) by combining rule-based and statistical methods, achieving high accuracy in text annotation and dictionary lookup, making ancient texts more accessible.

What Problem Does This Paper Address?

Sanskrit, an ancient and morphologically rich language, presents significant computational challenges. Its complex grammar, frequent compound formations, and the phenomenon of Sandhi (where sounds change at word boundaries) make automatic text processing difficult. Existing tools often struggle with accuracy or lack integrated functionality for both text analysis and dictionary access. This means scholars and enthusiasts often face a steep learning curve to search dictionaries or analyze texts, as words appear in inflected, compounded, or Sandhi-modified forms that don't directly match dictionary entries. The paper aims to bridge this gap by providing a system that can accurately parse these complex forms and link them to comprehensive dictionary information.

Methods

The core of the "Accessible Sanskrit" system is a cascading pipeline that prioritizes efficiency and accuracy. It starts with simple, deterministic methods and only escalates to more complex ones when necessary.

  1. Tier 1: Deterministic Matching:
    • Dictionary Lookup: The system first attempts to match words directly against a comprehensive database of Sanskrit dictionaries.
  2. Inflection Tables: For common inflected forms, it uses pre-computed inflection tables (derived from the Monnier-Williams dictionary) to map them to their root forms.
  3. Sandhi Rules: Simple, predictable Sandhi changes are handled with substitution rules.
  4. Tier 2: Statistical Approaches (if Tier 1 fails):
    • Scoring System: If deterministic methods don't yield a result, the system employs statistical approaches and evaluates the quality of potential splits using a scoring system based on length, morphology, and Sandhi application.
  5. Compound Splitting: For complex compounds, a quasi-brute-force splitter is used, trying various combinations and selecting the best-scoring result.
  6. Dictionary Integration: For every recovered entry, the system retrieves detailed grammatical information and dictionary entries from multiple sources.

The system also includes preprocessing steps like automatic transliteration detection (IAST) and handling of wildcard searches.

Datasets

  • GRETIL Corpus: Used for evaluating the system's accuracy on randomly selected compounds of various lengths.
  • Yoga Sūtra: Used as a practical use case for testing the system's performance on a complete text.
  • Monnier-Williams Dictionary: The primary source for building inflection tables and the core dictionary database.
  • Other Dictionaries: Concise Pali Dictionary, Grassmann, Apte Practical, Buddhist Hybrid, Cappeller, MacDonnell were also integrated to provide a richer dictionary resource.

The paper does not introduce new benchmarks but rather tests its system on existing corpora and discusses the limitations of current benchmarks like SandhiKosh.

Key Insights and Ideas

  • Cascading Approach is Key: Not all Sanskrit words are equally complex. A tiered approach, starting with fast, deterministic methods and only resorting to computationally intensive ones for difficult cases, is crucial for both efficiency and accuracy.
  • Prioritizing Dictionary Entry Retrieval: The system prioritizes returning complete dictionary entries for compounds and inflected forms, rather than just stemming them. This provides more contextually aware results for users.
  • Sandhi is the Toughest Nut to Crack: While the system achieves high accuracy overall, the Sandhi splitter is identified as the weakest component, suggesting it's the most challenging aspect of Sanskrit text processing.
  • Limitations of Existing Benchmarks: The paper highlights that current benchmarks often have specific conventions (e.g., splitting Sandhi but not compounds) that make them unsuitable for evaluating systems that handle both simultaneously.

Key Results

  • Accuracy on Compounds: The system achieves 90%+ accuracy on randomly selected compounds from the GRETIL corpus, regardless of compound length. This accuracy remains high even for compounds over 40 characters long.
  • Accuracy on Yoga Sūtra: A practical test on the complete Yoga Sūtra text demonstrated 96% accuracy.
  • Efficiency: The system is designed for practical performance, with the entire backend requiring only 2GB of RAM. Simple lookups are processed in milliseconds, while complex cases take at most 3 seconds on a virtual machine.
  • Accessibility: The system is available as an open-source Python library and a web application, making Sanskrit text analysis accessible to a wider audience.

Limitations and Future Work

  • Sandhi Splitter Weakness: The Sandhi splitter, despite scoring improvements, can still lead to incorrect splits for complex patterns.
  • Future Work: Replacing this component with neural models (like those described in Nehrdich et al., 2024) or fine-tuning LLMs (like Llora) is suggested to improve accuracy, though this may increase computational costs.
  • Dictionary and Inflection Table Coverage: The system's reliance on the Monnier-Williams dictionary means it can miss certain abstract words (e.g., with "-tva" suffix) and some common compounds. Inflection tables are also incomplete for irregular forms.
  • Future Work: Reconstructing inflection tables using a more comprehensive word list and potentially using language models to extract irregular forms from dictionaries and grammar books.
  • Stemming vs. Entry Retrieval: While prioritizing dictionary entry retrieval is an advantage for annotation, it's a limitation if the goal is pure stemming.
  • Future Work: The paper suggests that future versions could incorporate a ByT5-based model as a final step to handle the most complex cases, potentially improving performance.

Summary updated

BibTeX
@inproceedings{luca2025accessible,
  author = {Giacomo De Luca},
  title = {{Accessible Sanskrit: A Cascading System for Text Analysis and Dictionary Access}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure