Abstract
Standard Transformer models struggle with long-range dependencies due to the quadratic complexity of self-attention. This paper introduces TRANSYNC, a framework that enables efficient processing of long sequences by synchronizing information across segments via "anchor tokens." TRANSYNC achieves state-of-the-art results on complex reasoning benchmarks like NarrativeQA and HotpotQA while maintaining high computational efficiency and reduced memory overhead.
The Long-Range Problem in Transformers
Self-attention mechanisms in standard Transformers (e.g., BERT, GPT) scale quadratically ($O(l^2)$) with input length. This makes processing extensive documents or multi-document sets computationally prohibitive on standard hardware. Existing solutions typically follow two paths: sparse attention, which may miss distant connections, or hierarchical encoding, which often results in "shallow" information exchange where global context is only integrated at the final stages of processing.
The TRANSYNC Framework
TRANSYNC (Transformer with Synchronization) addresses these limitations by establishing "bridges" between segments using anchors. The process involves four key steps:
- Segmentation: Long inputs are divided into manageable chunks, each prefixed with the user's question to maintain local relevance.
- Anchor Identification: Specific tokens (e.g., document titles, special separators, or task-relevant entities) are identified as anchors across segments.
- Synchronization Module: Within every Transformer layer, a synchronization step occurs after local attention. Anchor tokens belonging to the same group (e.g., occurrences of the same title across different chunks) exchange information via a dedicated self-attention module.
- Iterative Propagation: By synchronizing at every layer, the model facilitates deep information flow, allowing a complex, global understanding to emerge incrementally as the text is processed.
Experimental Evaluation
The framework was evaluated on two challenging benchmarks designed to test long-range reasoning:
- NarrativeQA (Summary Setting): Requires answering questions based on long book and movie script summaries, testing single-document comprehension.
- HotpotQA (Wild Multi-hop Reasoning): A modified version where the model must find a chain of evidence across two relevant documents hidden among eight "distractor" documents.
Key Results and Efficiency
TRANSYNC consistently outperformed established baselines, including the strong Fusion-in-Decoder (FiD) and ETC methods.
As shown in the performance table, TRANSYNC achieves a consistent improvement (approximately 1% in Rouge-L and F1 scores) over competitive models. Beyond accuracy, the model demonstrates significant efficiency gains. On the NarrativeQA dataset, TRANSYNC was roughly 2x faster than the standard BART baseline, processing tokens in 134 $\mu$s compared to 292 $\mu$s. This efficiency stems from its ability to maintain a lower computational complexity ($O((l_q + \frac{l_c}{n})^2 \cdot n)$) compared to the standard quadratic approach.
Discussion and Future Work
The success of TRANSYNC highlights the importance of deep, layer-wise synchronization for long-context tasks. By using anchors as bridges, the model mimics human reading behavior—linking new information back to previously established entities.
While the current implementation relies on heuristic anchor selection (such as titles), future work could explore automated anchor discovery, where the model learns to identify the most critical "bridge" tokens dynamically. Additionally, scaling this synchronization mechanism to massive models with tens of billions of parameters remains an open and promising research direction.
Summary updated