Presented at EMNLP 2018, this research by Hayato Kobayashi challenges the conventional approach to combining machine learning models for text generation. While the field typically focuses on complex "runtime" integration, this paper demonstrates that a simple post-processing step—termed "post-ensemble"—is not only faster but often more effective for abstractive summarization tasks.
The Problem with Runtime Ensembles
In deep learning, ensembling is a reliable technique for boosting performance. For text generation tasks like summarization or translation, the standard approach is runtime-ensemble. This involves averaging the word-prediction probabilities of all constituent models at every single step of the beam search process.
However, runtime-ensemble has significant drawbacks:
- Computational Cost: Resource requirements scale linearly with the number of models, leading to high latency and memory usage.
- Implementation Complexity: It requires invasive modifications to the decoding logic of neural frameworks.
- Parallelization Hurdles: Models must synchronize at every generated token, making distributed execution difficult.
Methodology: The Post-Ensemble Approach
The proposed post-ensemble method treats models as independent "black boxes." Instead of integrating them during decoding, the algorithm follows a selection-based workflow:
- Generate: Run $M$ different models independently on the same input to produce $M$ candidate outputs.
- Embed: Convert these candidates into vectors using Bag-of-Words or pretrained word embeddings.
- Score: Calculate the average cosine similarity of each candidate to all other candidates in the set.
- Select: Choose the candidate with the highest average similarity—the "majority-like" output.
This similarity-based selection is mathematically grounded as an approximation of Kernel Density Estimation (KDE) using a von Mises-Fisher kernel, which is specifically designed for directional data in high-dimensional embedding spaces.
Experimental Setup
The method was evaluated using the Gigaword dataset, a standard benchmark for abstractive summarization where the task is to generate a news headline from an article's first sentence. The study utilized 3.8 million training pairs and tested against a suite of 128 pretrained models, which the author has released to the research community to facilitate further ensemble studies.
Key Results and Performance
The post-ensemble method demonstrated superior performance and scalability compared to traditional baselines. Using an ensemble of 32 models, the approach achieved a 37.52 ROUGE-1 score, outperforming both the state-of-the-art single model (37.27) and traditional runtime ensembles.
Key insights from the results include:
- Efficiency: The selection overhead was a mere 3.7 ms per sentence, negligible compared to the 44 ms required for the actual decoding.
- Scalability: Unlike runtime ensembles, which often encounter memory limits, post-ensemble scaled easily up to 128 models.
- Robustness: By selecting from existing valid outputs, the method provides a higher "quality floor" and is less susceptible to being dragged down by a single overconfident but incorrect model.
Conclusion
While post-ensemble is limited by its inability to "blend" parts of different sentences into a novel output, its implementation is "frustratingly easy" and highly effective. It offers a practical path for deploying large-scale ensembles in production environments where low latency and architectural simplicity are paramount. Future work may explore learning-to-rank approaches to further refine the selection process.
Summary updated