Abstract
We have tried to reproduce the results of the paper "Natural Language Inference over Interaction Space" submitted to ICLR 2018 conference as part of the ICLR 2018 Reproducibility Challenge. Initially, we were not aware that the code was available, so we started to implement the network from scratch. We have evaluated our version of the model on Stanford NLI dataset and reached 86.38% accuracy on the test set, while the paper claims 88.0% accuracy. The main difference, as we understand it, comes from the optimizers and the way model selection is performed.
Introduction
This report details an attempt to replicate the results of the "Densely Interactive Inference Network" (DIIN) for Natural Language Inference (NLI). While the researchers successfully built a high-performing model reaching 87.27% accuracy (close to the original 88.0%), their journey reveals how "hidden" implementation details—from optimizer schedules to framework defaults—can create significant hurdles for reproducibility in deep learning.
Methods
The authors re-implemented the DIIN architecture using Keras and TensorFlow. The DIIN pipeline consists of several sophisticated stages:
- Embedding Layer: Combines pre-trained word embeddings (GloVe), character-level features (via CNNs), and linguistic features like Part-of-Speech (POS) tags.
- Encoding Layer: Uses a variant of self-attention to capture context within each sentence.
- Interaction Layer: Creates a high-dimensional "interaction space" between the premise and hypothesis.
- Feature Extraction: Employs a DenseNet architecture (traditionally used in computer vision) to extract hierarchical patterns from the interaction space.
- Output Layer: A simple linear layer with softmax for final classification.
The reproduction effort involved significant "detective work," comparing their scratch-built code against the original authors' released source code to find discrepancies in dropout placement, L2 regularization, and specific DenseNet configurations, such as the omission of batch normalization.
Key Insights and Ideas
The Optimizer is a Hyperparameter
The authors found that the choice of optimizer was paramount. The original paper suggested switching from Adadelta to SGD, but the reproduction team found their best results by using a three-stage pipeline: starting with ADAM, switching to Adadelta when progress stalled, and finally finishing with SGD.
Structural "Dark Matter"
Several critical details were missing from the original paper but present in the code, such as the specific number of DenseNet blocks (3 vs. 2) and the fact that DIIN uses max-pooling instead of average-pooling within those blocks. The report also points out that researchers often rely on the default weight initializers of their chosen framework (e.g., TensorFlow vs. PyTorch). Because these defaults differ and change over time, they can lead to non-reproducible results if not explicitly stated. The authors suggest that papers should report the exact number of trainable parameters as a "checksum" for verification.
Key Results
The reproduction effort was largely successful, though it did not perfectly match the original reported figures. The team iteratively improved their model by uncovering undocumented architectural details and refining the training schedule.
As shown in the table above, the best reproduction reached 87.27% accuracy (achieved by using the custom three-optimizer schedule over 47 epochs). The 0.73% gap remaining between the reproduction and the original is attributed to subtle differences in model selection (how often the model is evaluated on the development set) and potential ensembling techniques used in the original work.
Limitations and Future Work
The authors acknowledge they did not have time to run experiments on the MultiNLI or Quora Question Pairs datasets. They also did not attempt ensembling (combining multiple models), which pushed the original paper's results even higher to 88.9%. Future work in this area involves standardizing how "minor" details—like evaluation frequency and specific framework versions—are reported to ensure that "state-of-the-art" results are truly accessible to the broader research community.
Summary updated