Database Architectures Tailored for Complex Linguistic Structures

16 concepts8 papers1 proposal

Standard database management systems (DBMS) are built on the assumption that data can be cleanly organized into flat, predictable, and normalized structures. In the relational model, information is carved up into tables where every row represents a record and every column contains an atomic value.

However, human language resists this flat geometry. Linguistic data—whether it is the deeply nested semantic structure of a dictionary entry or the hierarchical syntax of a parsed sentence—is inherently irregular, highly interconnected, and dynamic. Forcing language into traditional database schemas creates a fundamental architectural mismatch. To bridge this gap, researchers have developed specialized data models and querying techniques designed to handle the unique internal complexities of linguistic information.

The Mismatch Between Relational Tables and Linguistic Reality

When we try to store a dictionary or a lexicon in a standard relational database, we immediately run into the limits of database normalization. Normalization aims to eliminate redundancy by breaking data apart into isolated tables linked by foreign keys. But in a dictionary, a single word can have multiple nested senses, variable argument structures, and complex cross-references that defy atomicity.

As Nagao et al. argued in An Attempt to Computerized Dictionary Data Bases, general-purpose DBMSs are perfectly capable of handling the external structures of a database—such as how records are indexed and retrieved from disk—but they are entirely blind to the internal structures, meaning the intricate linguistic relationships contained within the data itself.

Standard Relational Model (Flat & Atomic):
[ Word ID ] -> [ Grapheme ] -> [ Part of Speech ]

Linguistic Reality (Nested & Interconnected):
[ Word ] ──> [ Morphological Roots ]
         ──> [ Sense 1 ] ──> [ Context A ] ──> [ Polyvalent Correlates ]
         │               ──> [ Context B ]
         ──> [ Sense 2 ] ──> [ Exceptions ]

Forcing these internal structures into rigid tables creates severe practical friction. In Denormalization and Cross Referencing in Theoretical Lexicography, Grimes demonstrated that representing polyvalent lexical correlates and non-unique mappings between semantic roles requires a deliberately denormalized approach. Strict normalization forces lexicographers to navigate a labyrinth of joined tables, making the database highly impractical for real-world fieldwork and data entry.

Furthermore, language is full of structural irregularities and exceptions. In A Feature-Based Model for Lexical Databases, Veronis et al. highlighted that nested word senses and idiosyncratic grammatical exceptions do not fit into uniform columns. Machine-readable dictionaries are often composed of highly unstructured definition texts. As Boguraev et al. pointed out in A Multi-Purpose Interface to an On-line Dictionary, retrieving knowledge from these texts requires flexible, heterogeneous querying mechanisms that standard database interfaces simply do not provide.

Moving Morphology Inside the Database

In a conventional database, a string like "running" is treated as an opaque, atomic blob of characters. The database has no native awareness that "running", "runs", and "ran" share the root "run". Traditionally, computer scientists solved this by writing external application logic to handle morphological analysis before passing queries to the database. However, this separation makes data maintenance fragile and inefficient.

To fix this, researchers proposed moving linguistic intelligence directly into the database engine. In Towards a Dedicated Database Management System for Dictionaries, Domenig et al. designed a dedicated DBMS that integrated morphological analysis into its core architecture. Instead of storing static, isolated strings, the database itself acts as a dynamic word analyzer.

Domenig expanded on this paradigm with the creation of Word Manager, detailed in Word Manager: A System for the Definition, Access and Maintenance of Lexical Databases. The author argued that relying on general-purpose databases forces developers to write massive amounts of indirect, custom procedural code to manage how words inflect based on grammatical context or how new words are formed. By utilizing a specialized formalism tailored for linguistic knowledge, a dedicated system can natively track dynamic word forms and morphological variations far more efficiently.

Scaling Structural Queries via Information Retrieval

The limitations of traditional databases become even more pronounced when moving from individual words to syntax. Treebanks—large collections of sentences annotated with their syntactic structure—are typically represented as deeply nested tree graphs.

If a linguist wants to query a treebank for a specific structural pattern (for example, finding all sentences where a noun phrase directly dominates a verb phrase containing a specific keyword), a relational database must perform a massive, computationally expensive sequence of table joins to traverse the parent-child relationships of the nodes. Even XML databases, which natively support hierarchical trees, fail to scale when these structural queries are run over millions of sentences.

Relational Approach (Slow):
[Node Table] ──(Join)──> [Edge Table] ──(Join)──> [Properties Table]  ==> High Latency

Information Retrieval Approach (Fast):
[Linearized Tree Token] ──> [Inverted Index] ──> [Instant Document Match] ==> Low Latency

To solve this bottleneck, Ghodke et al. introduced a novel optimization strategy in Fast Query for Large Treebanks. Instead of treating the treebank as a graph to be traversed via relational algebra, they reframed the problem using Information Retrieval (IR) techniques.

The authors treated each sentence as a "document" and linearized the hierarchical tree structures into specialized text tokens. By building an inverted index over these tokens, the system can evaluate complex structural and sequential constraints using rapid text-search algorithms rather than recursive database joins. The authors later scaled this methodology into Fangorn, a web-based system described in Fangorn: A System for Querying very large Treebanks. By combining the indexing power of text search engines with specialized linguistic query capabilities, they demonstrated that bypassing relational database constraints allows researchers to query massive syntactic datasets orders of magnitude faster than previously possible.

Key Foundational Work

An Attempt to Computerized Dictionary Data Bases (1980) — contributed 5 concept elements to this direction.
Denormalization and Cross Referencing in Theoretical Lexicography (1984) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_internal_vs_external_data_structures_fc3756d1,
  title = {internal vs. external data structures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-vs-external-data-structures/rdr_fc3756d13d044743976dcede191e97f5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_internal_vs_external_data_structures_fc3756d1_v1,
  title = {internal vs. external data structures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-vs-external-data-structures/rdr_fc3756d13d044743976dcede191e97f5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake