Exploiting Language Model for Efficient Linguistic Steganalysis: An Empirical Study

Biao Yi, Hanzhou Wu, Guorui Feng, Xinpeng Zhang

Abstract

Recent advances in linguistic steganalysis have successfully applied CNN, RNN, GNN and other efficient deep models for detecting secret information in generative texts. These methods tend to seek stronger feature extractors to achieve higher steganalysis effects. However, we have found through experiments that there actually exists significant difference between automatically generated stego texts and carrier texts in terms of the conditional probability distribution of individual words. Such kind of difference can be naturally captured by the language model used for generating stego texts. Through further experiments, we conclude that this ability can be transplanted to a text classifier by pre-training and fine-tuning to improve the detection performance. Motivated by this insight, we propose two methods for efficient linguistic steganalysis. One is to pre-train a language model based on RNN, and the other is to pre-train a sequence autoencoder. The results indicate that the two methods have different degrees of performance gain compared to the randomly initialized RNN, and the convergence speed is significantly accelerated. Moreover, our methods achieved the best performance compared to related works, while providing a solution for real-world scenario where there are more cover texts than stego texts.

Introduction

Linguistic steganography is the art of hiding secret information within natural language. While older methods modified existing text, modern generative steganography uses AI language models to create entirely new sentences that look perfectly normal but encode binary data in their word choices. Because these AI-generated "stego" texts are designed to mimic the statistical distribution of real "carrier" (normal) text, they are incredibly hard to detect. Standard deep learning classifiers often require massive amounts of labeled data and long training times to find the subtle "fingerprints" left behind by the data-embedding process. This study seeks a way to make steganalysis more efficient, especially in real-world scenarios where normal text is abundant but secret-carrying text is scarce.

Methodology

The core insight of this research is that stego texts, while natural-looking, deviate from the conditional probability distributions of normal language. To capture these nuances, the authors proposed a two-stage training framework involving pre-training and fine-tuning.

The proposed two-stage training framework for linguistic steganalysis.

Two pre-training strategies were explored:

  1. LM-RNN (Language Model RNN): The model is first trained to predict the next word in a sequence using a large corpus of normal carrier texts. This teaches the network the "ground truth" of how normal language flows.
  2. AE-RNN (Autoencoder RNN): The model is trained to encode a sentence into a vector and then reconstruct it. This forces the model to understand the global structure of a sentence.

After pre-training, the models are fine-tuned as binary classifiers to distinguish between Carrier and Stego texts. Starting with pre-trained weights rather than random initialization makes the model significantly more sensitive to the anomalies introduced by steganography.

The Perplexity Gap and Pre-training

A key finding is the "Perplexity Gap." Perplexity measures how "surprised" a language model is by a sequence of words. When a language model is trained on normal carrier texts, it finds those texts predictable (low perplexity). In contrast, stego texts—which must occasionally choose less probable words to encode data—result in higher perplexity.

Comparison of perplexity distributions between a randomly initialized RNN (a) and an RNN pre-trained on carrier texts (b).

As shown in the figure above, a randomly initialized model cannot distinguish between text types. However, once pre-trained on carrier texts, the model clearly identifies the higher perplexity of stego texts, especially at higher embedding rates (bits per word).

Key Results

The researchers evaluated their methods using the MOVIE and TWITTER benchmarks:

  • Superior Accuracy: The proposed LM-RNN and AE-RNN models consistently outperformed existing methods like CNNs and GNNs. On the MOVIE dataset (1.0 bits-per-word), the AE-RNN achieved a state-of-the-art accuracy of 96.18%.
  • Efficiency and Convergence: Pre-training acts as a shortcut. Pre-trained models reached lower loss levels and higher accuracy in a fraction of the epochs required by randomly initialized networks.
  • Global vs. Local Context: The AE-RNN generally outperformed the LM-RNN, suggesting that learning the entire sentence structure (global information) is more effective for steganalysis than just predicting the next word (local information).
  • Addressing Data Imbalance: This method allows leveraging vast amounts of unlabeled normal text to improve detectors, providing a practical solution for real-world scenarios where stego samples are rare.

Conclusion

This study demonstrates that pre-training language models or autoencoders on carrier text significantly enhances the performance and efficiency of linguistic steganalysis. While highly effective on current benchmarks, future work will explore scaling these insights to larger Transformer-based models and testing robustness against more sophisticated steganographic algorithms.

Summary updated

Abstract

Recent advances in linguistic steganalysis have successively applied CNN, RNN, GNN and other efficient deep models for detecting secret information in generative texts. These methods tend to seek stronger feature extractors to achieve higher steganalysis effects. However, we have found through experiments that there actually exists significant difference between automatically generated stego texts and carrier texts in terms of the conditional probability distribution of individual words. Such kind of difference can be naturally captured by the language model used for generating stego texts. Through further experiments, we conclude that this ability can be transplanted to a text classifier by pre-training and fine-tuning to improve the detection performance. Motivated by this insight, we propose two methods for efficient linguistic steganalysis. One is to pre-train a language model based on RNN, and the other is to pre-train a sequence autoencoder. The results indicate that the two methods have different degrees of performance gain compared to the randomly initialized RNN, and the convergence speed is significantly accelerated. Moreover, our methods achieved the best performance compared to related works, while providing a solution for real-world scenario where there are more cover texts than stego texts.
BibTeX
@inproceedings{yi2022exploiting,
  author = {Biao Yi and Hanzhou Wu and Guorui Feng and Xinpeng Zhang},
  title = {{Exploiting Language Model for Efficient Linguistic Steganalysis: An Empirical Study}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746219},
}
Figure