SMIC: Shared Multiscale Input-Reference Conditioning across Transformer Layers for Natural Language Inference

Xin Zhang, Victor S. ShengarXiv 2026 · arXiv

TL;DR

SMIC (Shared Multiscale Input-Reference Conditioning) is a novel architectural modification that gives every layer in a Transformer direct, multiscale access to the original input. By injecting these "refresher" features before each attention update, the authors achieved state-of-the-art accuracy on the challenging Adversarial NLI (ANLI) benchmark while significantly reducing the model's sensitivity to small, localized word changes.

What Problem Does This Paper Address?

Natural Language Inference (NLI)—the task of determining if a premise entails, contradicts, or is neutral toward a hypothesis—remains a significant hurdle for AI. Even massive models can be surprisingly brittle; changing a single word (a "localized lexical change") can flip a prediction, even when the underlying logic remains the same.

Traditional Transformers process information hierarchically, but as data moves through deeper layers, the fine-grained details of the original input can become "washed out" or overly abstracted. While scaling models up often helps, it comes with massive computational costs and diminishing returns. The authors of this paper sought a more efficient architectural solution: how can we keep the model "grounded" in the original input across all its layers without just throwing more parameters at the problem?

Methods

The core innovation is SMIC (Shared Multiscale Input-Reference Conditioning). Here’s how it works:

  1. Shared Reference: The model builds a single "reference" representation from the input embeddings using convolutional filters of various widths (multiscale). This captures both individual token information and local context (spans of words).
  2. Layer-Specific Injection: Instead of only seeing this at the beginning, every Transformer layer gets access to this shared reference. However, each layer applies its own unique, learned transformation to the reference before injecting it into the hidden state.
  3. TCT-SMIC: The authors integrate SMIC into a Token–Context Transformer (TCT). This architecture uses complex-vector representations where the real and imaginary parts represent "token" and "context" roles, respectively.
  4. Attention Diagnostics: To measure robustness, they used Jensen–Shannon Divergence (JSD) to quantify how much a model's attention "redistributes" when the input is slightly perturbed. They even established a mathematical bound proving that lower JSD directly correlates to more stable attention.

Datasets

The primary evaluation target was ANLI (Adversarial NLI), which consists of three rounds (R1, R2, R3) of increasingly difficult examples designed to fool state-of-the-art models.

For diagnostic testing, they used:

  • WordNet: To generate synonym-based perturbations.
  • Random Lexical Replacements: As a "stress test" for unconstrained corruption.
  • Pretrained Backbones: They also tested SMIC as a plug-in for ModernBERT and Llama-3 to prove its generalizability beyond their custom TCT architecture.

Key Insights and Ideas

  • Input "Grounding" is Efficient: You don't need a 10B parameter model to handle complex NLI if your architecture ensures deep layers don't lose track of the raw input.
  • Multiscale > Token-level: Ablation studies showed that building features from word spans (multiscale) is significantly more effective than simply re-injecting raw token embeddings at every layer.
  • Stability is Measurable: By proving a finite-sample bound on attention redistribution, the authors provide a theoretical framework for why SMIC makes models more robust: it prevents the "attention mechanism" from panicking and shifting wildly when a single word changes.
  • Plug-and-Play Potential: SMIC isn't just for niche architectures; it showed clear benefits when added to standard real-valued models like Llama, narrowing the gap in performance under adversarial conditions.

Key Results

TCT-SMIC consistently outperformed parameter-comparable baselines (like Phi-1.5, Llama-3-style, and Pythia-style models) across all three ANLI rounds.

Table 1 ANLI accuracy and training FLOPs across parameter-comparable configurations.

As shown in Table 1, TCT-SMIC (0.80B) reached 39.03% on R1, 44.89% on R2, and 43.17% on R3. Notably, it achieved these results with fewer training FLOPs (floating-point operations) than most baselines, indicating high computational efficiency.

Ablation tests confirmed that the SMIC module was a primary driver of these gains, particularly in the larger R2 and R3 datasets.

Table 2 Mean ANLI accuracy with and without SMIC.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Label Preservation: The automated perturbations (like random word swaps) weren't manually re-annotated. It's possible some changes actually should change the NLI label, meaning the "accuracy drop" measured might occasionally be the model being correct.
  • Architectural Isolation: Because the baselines used different tokenizers and internal configurations, it's hard to perfectly isolate the impact of SMIC versus other TCT features, though the Llama-style plug-in experiments help mitigate this concern.
  • Causality: The attention diagnostics are descriptive. While they show that the model is more stable, they don't definitively prove the causal chain of why that stability leads to higher accuracy.

Future work could involve applying SMIC to even larger pretrained models or exploring its utility in other domains sensitive to local perturbations, such as legal or medical document analysis.

Summary updated

Abstract

Natural Language Inference models can be sensitive to localized lexical changes. We introduce Shared Multiscale Input-Reference Conditioning (SMIC), which constructs a single input-dependent multiscale reference and reuses it across Transformer layers. Each layer applies its own multiscale transformation to the shared reference and injects the resulting conditioning feature before the attention update. We instantiate SMIC in a token--context Transformer (TCT) and conduct from-scratch comparisons against decoder-based and Multi-Layer Perceptron (MLP) baselines across all three ANLI rounds. TCT-SMIC attains the highest observed ten-seed mean accuracy on all three ANLI rounds. The ablations further support the contribution of SMIC. We evaluate a real-valued Llama-style plug-in and extend deletion/insertion diagnostics to pretrained ModernBERT encoder and Llama decoder models with and without SMIC. Across the evaluated perturbation protocols, TCT-SMIC exhibits comparatively low pooled-attention redistribution and smaller signed accuracy changes. We further establish a finite-sample bound on average subset-mass redistribution in terms of the mean Jensen--Shannon divergence between the restricted and renormalized attention distributions.
BibTeX
@misc{zhang2026smic,
  author = {Xin Zhang and Victor S. Sheng},
  title = {{SMIC: Shared Multiscale Input-Reference Conditioning across Transformer Layers for Natural Language Inference}},
  year = {2026},
  eprint = {2611.16270},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16270},
}
Figure