Abstract
The study of generalization in Language Models (LMs) requires controlled experiments that can precisely measure complex linguistic variations between training and testing datasets. We introduce DECAF, a framework that enables the analysis and filtering of linguistically annotated datasets down to the character level. Rather than creating new resources for each experiment, DECAF starts from datasets with existing linguistic annotations and leverages them to analyze, filter, and generate highly controlled and reproducible experimental settings targeting specific research questions. We demonstrate DECAF's functionality by adding 28 morphosyntactic annotation layers to the 115M-word BabyLM corpus and indexing the resulting 1.1B annotations to analyze its internal domain variance and to create a controlled training data curriculum for a small-scale gender bias study. We release DECAF as an open-source Python library, along with the parsed and indexed version of BabyLM, as resources for future generalization research.
Introduction
To understand how Language Models (LMs) generalize, researchers often perform "data interventions": they deliberately remove or reorder specific linguistic structures in the training data to see how the model reacts. Currently, this process is often cumbersome, relying on complex regular expressions (regex) to filter raw text files. This approach fails when filtering for hierarchical or nested structures, such as sentences where a specific pronoun acts as a direct object for a particular verb. Furthermore, these custom scripts lack reproducibility and extensibility, making it difficult for the community to build upon previous data-centric research.
Methodology: The Database as Corpus
DECAF (Dynamically Extensible Corpus Analysis Framework) addresses these issues by treating a corpus as a structured relational database rather than a flat text file. Using SQLite, DECAF organizes data into four primary tables: Literals (atomic units like characters), Structures (linguistic units like POS tags), Structure_Literals (mapping annotations to text spans), and Hierarchies (storing relationships between structures).
This architecture allows for complex queries via a Python API, enabling "non-linear" filtering based on syntactic relationships rather than mere string adjacency. The framework also supports sharding, allowing it to scale to massive datasets while maintaining a unified query interface. In testing, DECAF indexed the 115M-word BabyLM corpus (1.1B annotations) in just 1.5 hours on a standard laptop.
Experimental Results: Bias Dynamics and Domain Divergence
The authors demonstrated DECAF's utility by adding 28 morphosyntactic annotation layers to the BabyLM corpus and conducting a gender bias study using the WinoBias benchmark. A key finding concerned "recency bias" in training: even when the total training data was perfectly balanced (50/50 male and female pronouns), the order of presentation mattered significantly. Models tended to adopt the bias of the data they encountered most recently in the curriculum.
Additionally, DECAF enabled granular domain analysis. By calculating Jensen-Shannon Divergence, the authors found that Wikipedia and Project Gutenberg are significant outliers compared to spoken-word datasets like CHILDES or Switchboard. For instance, Wikipedia almost never uses the first person ("I"), while spoken data is saturated with it.
Conclusion and Future Work
DECAF provides a powerful, reproducible tool for data-centric LM research, allowing researchers to replace "regex nightmares" with clean, readable Python objects. While it currently depends on external parsers and primarily supports CoNLL-U formats, it represents a significant step toward more surgical and controlled training data interventions. Future work will focus on expanding the API to handle even more complex recursive queries natively.
Summary updated