Abstract
We present a self-contained system for constructing natural language models for use in text compression. Our system improves upon previous neural network based models by utilizing recent advances in syntactic parsing -- Google's SyntaxNet -- to augment character-level recurrent neural networks. RNNs have proven exceptional in modeling sequence data such as text, as their architecture allows for modeling of long-term contextual information.
Introduction
Data compression is fundamentally a two-step process: modeling and coding. While coding—the conversion of probabilities into bits—is largely a solved problem through techniques like arithmetic coding, modeling remains the frontier of research. Effective text compression requires a model capable of accurately predicting the next character in a sequence; higher prediction accuracy directly translates to fewer bits required for storage.
Traditional models often fail to capture long-term dependencies or the complex inherent structure of human language. Previous neural network approaches frequently treated text as a raw stream of symbols, remaining "blind" to syntax. This research addresses this gap by investigating whether a compression model can be made more efficient by incorporating the grammatical role of each character. By augmenting character-level Gated Recurrent Units (GRUs) with Part-of-Speech (POS) tags from Google’s SyntaxNet, we demonstrate a significant boost in next-character prediction accuracy.
Methodology
The proposed "Syntactically Informed" architecture utilizes Gated Recurrent Units (GRUs), a type of Recurrent Neural Network (RNN) designed to maintain information over long sequences while avoiding the vanishing gradient problem.
The core of the system is a dual-input pipeline:
- Character Stream: A one-hot encoded representation of a 40-character sliding window.
- Syntactic Stream: Parallel to the character input, the model receives POS tags (e.g., noun, verb, punctuation) generated by Google’s SyntaxNet.
As shown in the architecture diagram, these two streams are processed by separate GRU layers before being merged (concatenated). The merged representation passes through a final GRU and two dense (fully connected) layers. The output is a Softmax distribution representing the probability of the 256 possible ASCII characters. This probability distribution is then utilized by an arithmetic coder to perform the actual file compression.
Experimental Setup
The models were trained on five classic literary texts sourced from Project Gutenberg. These documents vary in length, providing a range of data scales for testing the model's performance and its ability to generalize across different writing styles.
The dataset includes Alice in Wonderland (the shortest), The Adventures of Sherlock Holmes, The Complete Works of Friedrich Nietzsche, Pride and Prejudice, and A Tale of Two Cities (the longest).
Results and Discussion
Grammar as a Predictive Guide
The primary insight from this work is that linguistic metadata, specifically POS tags, provides a critical shortcut for the neural network. Rather than forcing the model to infer grammatical structure solely from character sequences, the POS tags explicitly define the grammatical category of the upcoming sequence. This significantly narrows the search space for the next character, improving prediction efficiency.
Overfitting as a Compression Strategy
In typical machine learning contexts, overfitting is avoided. However, for document-specific compression, memorization is the goal. If a model perfectly memorizes a specific text, it can predict every character with 100% certainty, theoretically allowing the file to be compressed to near zero bytes (excluding the model's own size). On shorter documents like Alice in Wonderland, the model achieved near-perfect accuracy, indicating successful memorization.
Performance Gains
The inclusion of syntactic information yielded measurable improvements. For the Pride and Prejudice model, adding POS tags resulted in an average accuracy increase of 5.33%. Furthermore, while models were document-specific, a model trained on Pride and Prejudice maintained approximately 54-59% accuracy when tested on entirely different books. This suggests that a generalized pre-trained model could potentially serve as a viable foundation for general-purpose compression.
RNN Dynamics and Stability
The study also highlighted the complexity of RNN training. In the longest text, A Tale of Two Cities, the model exhibited "chaotic behavior" where accuracy plummeted after 650 epochs due to gradient instability. This underscores that even advanced architectures like GRUs require careful tuning when processing very large datasets.
Conclusion and Future Work
While the results are promising, several limitations remain. The current approach is computationally intensive, requiring approximately 48 hours of training per document on high-end hardware, making it impractical for real-time applications. Additionally, the one-hot encoding scheme is not memory-efficient.
Future research will explore moving beyond simple POS tags to incorporate word dependency trees. By capturing the relationships between distant words (such as a subject and its verb), the model could potentially identify even deeper contextual patterns, leading to further improvements in compression ratios.
Summary updated