Competing Independent Modules for Knowledge Integration and Optimization

Parsa Bagherzadeh, Sabine BerglerEMNLP 2021 · DOI

Abstract

This paper introduces a flexible neural framework that shifts away from monolithic "black box" models toward a modular architecture. By treating diverse knowledge sources—including sentiment lexica, part-of-speech tags, and individual Transformer layers—as independent, competing components, we demonstrate a robust and interpretable method for enhancing NLP performance. The proposed Interacting Independent Module (IIM) framework allows for the integration of inconsistent external data and improves standard BERT performance by treating its internal layers as independent modules, all without the need for expensive end-to-end retraining.

Introduction

Modern NLP is dominated by large, pre-trained language models (LMs) like BERT. While powerful, these systems are typically monolithic, making it difficult to inject domain-specific knowledge (e.g., specialized medical lexica or grammatical rules) without expensive fine-tuning or complex architectural changes. Furthermore, when multiple external knowledge sources are available, they often conflict or introduce noise, leading to "interference" that can degrade the learning process.

We address these challenges by proposing a "plug-and-play" framework where independent neural modules represent different knowledge sources and compete for attention via a central controller. This approach is designed to be robust to inconsistent information, flexible enough to add or remove sources without retraining, and transparent in its activation of specific knowledge components.

Methods: The Interacting Independent Module Framework

The core of our approach is the Interacting Independent Module (IIM) framework. Rather than a single large network, the system comprises three main components:

  1. Independent Modules ($R_n$): Each module is a recurrent neural network (such as an LSTM or GRU) dedicated to one specific knowledge source, such as part-of-speech (POS) tags or a sentiment lexicon.
  2. The Controller ($C$): An LSTM-based "manager" that looks at the input and decides which $k$ modules should be active at any given time step. This creates a competitive environment where only the most relevant modules for a specific word are updated.
  3. Interaction Mechanism: Active modules use an attention mechanism to "read" information from all other modules, allowing for a collaborative exchange of features.
The Interacting Independent Module framework architecture, illustrating the controller's selection of active modules over time steps.

This framework can also be applied to Transformer Layer Integration. Instead of passing data through BERT layers sequentially, we treat each layer as an independent module. This allows the model to simultaneously access low-level "surface" features from early layers and high-level "semantic" features from later layers.

Experimental Results

We evaluated the framework on several standard sentiment analysis benchmarks, including SST-2, SemEval 2017 (SE17-4), and aspect-based sentiment tasks (SE14-L, SE14-R, SE15-11). Our modular approach consistently outperformed baseline BERT and RoBERTa models across all tasks.

Performance comparison of BERT and RoBERTa across multiple benchmarks with various module configurations.

As shown in the results, integrating a full suite of modules (Token + POS + Dependencies + Sentiment Lexica) provided significant gains. For example, on the SE14-L task, the modular approach improved the macro-F1 score from 72.1 to 77.2. Notably, using a frozen BERT model combined with modular integration often outperformed a standard fine-tuned BERT, offering a major win for computational efficiency.

Key Insights and Discussion

  • Competition Breeds Specialization: Limiting the number of active modules ($k < N$) often led to better performance than using all modules at once. This sparsity forces modules to specialize on specific types of tokens or linguistic patterns.
  • Layer-Wise Activation: When treating Transformer layers as modules, the controller frequently activated the first two layers (lexical triggers) and the seventh layer (grammatical relations), confirming theories that different layers specialize in distinct linguistic tasks.
  • Robustness to Inconsistency: Because backpropagation is spread independently over modules, the system can weigh the usefulness of conflicting lexica, effectively ignoring a source if it does not contribute to the specific task.

Conclusion

The Interacting Independent Module framework provides a scalable and interpretable alternative to monolithic LMs. By fostering competition among specialized modules, the system achieves state-of-the-art performance while maintaining computational efficiency and transparency. Future work will explore the application of this modularity to generative tasks and the dynamic adjustment of module activation based on input complexity.

Summary updated

BibTeX
@inproceedings{bagherzadeh2021competing,
  author = {Parsa Bagherzadeh and Sabine Bergler},
  title = {{Competing Independent Modules for Knowledge Integration and Optimization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.376},
}
Figure