Abstract
In Natural Language Processing (NLP), modeling the relationship between two sequences—sequence-to-sequence (seq2seq) mapping—is a fundamental challenge. Traditional approaches relied on complex pipelines of hand-crafted features and external resources. This research advances the state-of-the-art by exploring neural models for sequence representation, modeling interactions between sequences, and leveraging unpaired data. We introduce a novel "segment-to-segment" transduction model for online generation and a noisy channel framework that utilizes massive amounts of unpaired data to improve translation and summarization.
1. Introduction
Sequence-to-sequence mapping is central to tasks like machine translation, summarization, and question answering. Historically, these problems were addressed using modular systems requiring significant domain expertise and manual feature engineering. While early neural encoder-decoder models simplified these pipelines, they introduced a "bottleneck" by requiring the entire input to be compressed into a fixed-length vector. Furthermore, these models are typically data-hungry, requiring large parallel corpora that are often unavailable. This thesis addresses these limitations through three primary innovations: distributed representations, segment-based transduction, and the integration of unpaired data via a noisy channel model.
2. Distributed Sentence Representations
For tasks such as Answer Sentence Selection, we investigate neural architectures to encode sentences into semantic vectors. By comparing Bag-of-Words, Convolutional Neural Networks (CNNs), and Long Short-Term Memory (LSTM) networks, we evaluate how different architectures capture semantic meaning. These vectors are matched using a bilinear model to determine the relevance of a candidate answer to a given question. Our findings indicate that LSTMs, particularly when combined with attention mechanisms, significantly outperform simpler models by capturing long-range dependencies and word order.
3. Segment-to-Segment Neural Transduction (SSNT)
A major architectural contribution of this work is the Segment-to-Segment Neural Transduction (SSNT) model. Unlike standard encoder-decoders that process the entire input before generating output, SSNT employs a latent variable to decide between two actions: SHIFT (reading more input) or EMIT (generating an output token). This architecture allows for "online" generation, where the model begins producing output before the full input is processed—mimicking human simultaneous interpretation. By processing segments, the model avoids the fixed-length vector bottleneck and can achieve high performance with smaller hidden layers.
4. The Neural Noisy Channel Model
To address the scarcity of parallel data, we reformulate the seq2seq problem using a noisy channel framework based on Bayes’ rule: $p(y|x) \propto p(x|y)p(y)$. This approach decomposes the problem into a "channel model" (mapping output back to input) and a "source model" (a language model). The primary advantage is that the source model can be trained on vast amounts of unpaired, monolingual data in the target language. This provides a powerful prior for fluency and significantly boosts performance in translation and summarization tasks where parallel data is limited.
5. Experimental Evaluation
The proposed models were evaluated across several NLP benchmarks, including TREC QA and WikiQA for answer selection, the Gigaword dataset for abstractive summarization, and FBIS/IWSLT datasets for machine translation.
Key Results
- State-of-the-Art Performance: Neural matching models achieved new SOTA results on TREC QA and WikiQA, surpassing complex feature-engineered systems.
- Efficiency in Summarization: The SSNT model outperformed vanilla encoder-decoders and matched the performance of much larger attention-based models.
- Leveraging Unpaired Data: In Chinese-English translation, the noisy channel model improved performance by approximately 3 BLEU points by effectively utilizing 4.3 million unpaired English sentences.
- Grammatical Learning: The models achieved over 97% accuracy on German verb inflections, demonstrating an ability to learn complex morphological rules directly from data.
6. Conclusion and Future Directions
This research demonstrates the power and flexibility of neural networks for seq2seq mapping. While the SSNT model is currently limited by a monotonicity constraint—assuming input and output sequences move in the same direction—it paves the way for real-time applications like live captioning. Future work will focus on relaxing this constraint for language pairs with divergent word orders, reducing the computational complexity of alignment, and scaling these models to handle document-level processing.
Summary updated