Integrating Discrete and Neural Features Via Mixed-Feature Trans-Dimensional Random Field Language Models

Silin Gao, Zhijian Ou, Wei Yang, Huifang Xu

Abstract

This paper introduces a "mixed-feature" Trans-dimensional Random Field (TRF) language model that natively integrates both discrete (n-gram) and neural (LSTM) features into a single framework. By combining these complementary strengths, the model achieves state-of-the-art speech recognition performance while offering faster training convergence and significantly more efficient inference than standard LSTM models.

Introduction

In language modeling (LM), a persistent challenge has been effectively combining two distinct types of features: discrete n-grams, which excel at capturing local symbolic interactions, and neural features (like LSTMs), which are superior at modeling long-range dependencies and continuous smoothing. Traditionally, these are combined via interpolation—a sub-optimal two-step process where separate models are trained and then blended. This paper proposes a unified framework to learn both feature types simultaneously within a single model.

Methodology

The authors utilize the Trans-dimensional Random Field (TRF) framework, an undirected graphical model that assigns probabilities to entire sentences rather than predicting word-by-word.

The Mixed-Feature Potential Function

The core innovation is a potential function defined as the sum of a Linear Potential (using discrete word and class n-grams) and a Nonlinear Potential (using neural features from a multi-layer bidirectional LSTM).

Training with Dynamic Noise-Contrastive Estimation (DNCE)

The model is trained using DNCE, which treats training as a classification task: distinguishing real sentences from "noise" sentences. The noise distribution is updated dynamically alongside the main model to ensure robust learning.

Experimental Setup

The approach was evaluated on two benchmarks:

  • Penn Treebank (PTB): Used for detailed comparison (~1 million words).
  • Google One-Billion-Word Dataset: Used to demonstrate scalability (~0.8 billion words).

The models were evaluated on a rescoring task using the 1000-best hypotheses from a speech recognition system on the WSJ’92 test set.

Results and Discussion

The Mixed-feature TRF demonstrates significant advantages in both accuracy and efficiency over traditional standalone and interpolated models.

Performance and Efficiency

On the PTB dataset, the Mixed-feature TRF achieved a Word Error Rate (WER) of 7.17%, outperforming standalone LSTMs (7.36%) and standalone neural TRFs (7.34%). Notably, the model is significantly faster during inference—ranging from 5x to 114x faster than standard LSTMs—because it avoids the per-word Softmax bottleneck.

Comparison of model performance and efficiency on the PTB dataset.

Training Convergence

A surprising "concavity bonus" was observed: adding discrete features stabilizes the optimization process. Because the log-likelihood of discrete features is mathematically concave, it helps the overall non-convex neural network parameters converge faster than a pure neural model.

WER vs. training epochs for different TRF configurations, showing the faster convergence of the Mixed TRF.

Scalability

On the Google Billion Word dataset, the Mixed TRF reached a WER of 5.28%, a substantial improvement over the KN5 n-gram baseline (6.13%) and the LSTM baseline (5.55%). The mixed model matched the performance of the best interpolated models but required only a single training step, reducing overall pipeline complexity.

Conclusion

The Mixed-feature TRF provides a unified, efficient, and high-performing alternative to traditional interpolated language models. Future work will focus on moving beyond rescoring to integrate these models into "one-pass decoding" and exploring automated feature discovery to further reduce manual engineering.

Summary updated

BibTeX
@inproceedings{gao2020integrating,
  author = {Silin Gao and Zhijian Ou and Wei Yang and Huifang Xu},
  title = {{Integrating Discrete and Neural Features Via Mixed-Feature Trans-Dimensional Random Field Language Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9052929},
}
Figure