Gated Convolutional Sequence to Sequence Based Learning for English-Hingilsh Code-Switched Machine Translation.

Suman Dowlagar, Radhika MamidiNAACL 2021

TL;DR

This paper tackles the challenge of translating code-switched text, specifically English-Hinglish, by proposing a novel gated convolutional sequence-to-sequence model. Their approach leverages convolutions for better contextual understanding and gating mechanisms for improved representation, outperforming traditional RNN and Transformer models on this complex task.

What Problem Does This Paper Address?

Code-switching, the practice of embedding linguistic units from multiple languages within a single sentence, is prevalent in multilingual communities, especially on social media. This phenomenon poses significant challenges for natural language processing (NLP) tasks, including machine translation, due to variations in grammar, spelling, and informal language use. Existing machine translation models often struggle with the inherent complexities of code-switched data, necessitating specialized approaches to accurately translate such content.

Methods

The authors propose a gated convolutional sequence-to-sequence (ConvSeq2Seq) model for English-Hinglish machine translation. This model consists of an encoder and a decoder, both utilizing convolutional layers.

  • Encoder: The encoder processes the input English sentence. Each token is first embedded, and positional embeddings are added to retain sequence order, as convolutional models lack inherent recurrence. These embeddings are then fed through a series of gated convolutional blocks. The gated linear units (GLU) activation function and residual connections are employed to facilitate learning. The convolutional filters slide across the sequence, capturing local contextual features.
  • Decoder: The decoder mirrors the encoder's architecture but incorporates attention mechanisms. It uses the encoder's convolved outputs and attention to generate the translated Hinglish sentence. A feed-forward layer is used at the end to match the target dimension.

The core novelty lies in the application of gated convolutional networks for code-switched machine translation, aiming to capture compositional structures and contextual information more effectively than recurrent or standard attention-based models.

Datasets

The paper uses a Hinglish code-mixed dataset for training, validation, and testing. This dataset consists of 8060 training examples, 952 validation examples, and 960 test examples. The task is to translate English sentences into code-mixed Hinglish. The paper also references previous work that has created parallel corpora for Hindi-English code-switched scenarios.

Key Insights and Ideas

  • Convolutions for Context: The paper highlights that convolutional filters, with their sliding window mechanism, are effective at capturing local contextual information within code-switched sentences, which is crucial for understanding mixed-language structures.
  • Gating for Representation: The use of gated units (GLU) in the convolutional blocks helps in controlling the flow of information, leading to richer and more discriminative representations of the input text.
  • Code-Switching Complexity: The authors emphasize that code-switched text is inherently complex due to non-standard variations, making it a challenging but important area for NLP research.
  • Short Sentences vs. Long Sentences: The model performs better on shorter sentences, while longer sentences sometimes exhibit repetition in translations, leading to a decrease in BLEU scores. This is attributed to potential issues with low dependencies and a higher proportion of out-of-vocabulary words in longer, more complex code-switched utterances.

Key Results

The proposed gated convolutional sequence-to-sequence model achieved a BLEU score of 2.58 on the English-Hinglish translation task. This performance was superior to baseline models:

  • Encoder-Decoder RNN model: 1.52
  • Vanilla Transformer model: 2.51

The authors also note that their convolutional model was able to detect small-length sequences and code-switching points more effectively.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the limited size of the dataset might be a factor affecting performance, particularly for longer sentences and the presence of out-of-vocabulary words. They suggest that increasing the dataset size could improve translation accuracy.
  • Repetitions in Long Sentences: The model sometimes produces repetitive translations for longer sentences, which negatively impacts the BLEU score.
  • Future Work: The authors plan to explore pre-trained BERT models and their ensembles for code-switched machine translation. They also aim to incorporate other code-mixing factors and advanced pre-processing techniques to further enhance translation quality.

Summary updated

BibTeX
@inproceedings{dowlagar2021gated,
  author = {Suman Dowlagar and Radhika Mamidi},
  title = {{Gated Convolutional Sequence to Sequence Based Learning for English-Hingilsh Code-Switched Machine Translation.}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure