FastJEV: Understanding Redundancy for Compact JEV Inference

Jie Ma, Jie Gao, Yihang Liu, Zhike Qiu, Junle Li, Chongyi Zhuang, Jiayi Ji, Xiaoshuai SunarXiv 2026 · arXiv

TL;DR

FastJEV optimizes Joint Evaluation (JEV) models—which score multiple decision candidates directly—by eliminating redundant computation through shared state anchoring, prefix sharing, and decision-guided pruning. It reduces candidate execution depth by approximately 45% while retaining over 93% of original task performance without requiring any additional training.

What Problem Does This Paper Address?

Multimodal models are increasingly used for embodied AI and decision-making tasks where they must choose the best action from a set of candidates. Unlike standard generative models that produce text token-by-token, Joint Evaluation (JEV) models score a set of explicit alternatives (e.g., "pick up the red mug" vs. "pick up the blue bottle") based on a shared visual and textual context.

However, current JEV inference is inefficient. Even if the context is encoded once, evaluating $N$ different candidates often involves repeating matching token histories, duplicating inference states, and running the entire model backbone for every single choice. This redundancy leads to high latency and memory usage, especially as the number of candidates or the model size grows.

Methods

The authors propose FastJEV, a framework that organizes candidate evaluation into a compact structure using three key components:

  1. Shared Context Anchoring (S): Instead of copying the recurrent states of the shared context for every candidate branch, FastJEV uses a single read-only buffer. It also omits the allocation of final recurrent caches for "leaf" nodes that don't have further branches, saving memory.
  2. Candidate Prefix Sharing (T): Candidates often share long prefixes (e.g., "The robot is moving toward the..."). FastJEV organizes these into a trie (prefix tree), ensuring that shared token sequences are processed by the language backbone only once.
  3. Decision Guided Pruning (P): Not all layers contribute equally to the final decision. The authors use a small, unlabeled calibration set to measure "decision sensitivity"—how much skipping a layer changes the relative ranking of candidates. They then skip the least sensitive layers during candidate evaluation while keeping the full backbone for the initial context encoding.
Overview of the FastJEV architecture, showing how context anchoring, prefix sharing, and pruning work together.

Datasets

FastJEV was evaluated across three sizes of the OmniJev model (0.8B, 2B, and 4B parameters) using:

  • Public Benchmarks: AI2D (diagrams), ScienceQA-IMG (science questions), POPE (object hallucination), BLINK (visual perception), and ERQA (embodied reasoning).
  • LIBERO-10: A reconstructed offline question set based on robot manipulation demonstrations to test embodied decision-making.
  • Calibration: A small subset of OK-VQA was used for the offline pruning sensitivity assessment.

Key Insights and Ideas

  • The Trie Paradox: Interestingly, while candidate prefix sharing (T) significantly reduces the number of logical tokens processed, it can sometimes increase latency. This is because frequent branching creates many small segments, leading to overhead from multiple kernel launches and complex state management.
  • Non-Uniform Sensitivity: Pruning sensitivity does not decrease linearly with depth. As shown in Figure 2, the middle-to-late layers often show the lowest sensitivity, while the very final layer is critical for maintaining decision quality.
  • Ranking vs. Accuracy: The core insight of decision-guided pruning is that we don't need to preserve the exact logit values, only the relative order of candidates. This allows for much more aggressive pruning than standard task-agnostic methods.
Sensitivity curves across different model sizes, highlighting that redundancy is concentrated in specific layers.

Key Results

  • Efficiency: FastJEV reduces candidate execution depth by 43.75% to 45.83%.
  • Performance Retention: Across six evaluation sets, the method retains 93.66% to 97.52% of the original model's accuracy.
  • Speed and Memory: The "S+P" configuration (Anchoring + Pruning) achieved speedups of 1.05x to 1.26x on an NVIDIA L20 GPU. On the LIBERO robot benchmark, peak memory usage was reduced by up to 15.7%.
  • Qualitative Success: The authors demonstrate that even with significant pruning, the model correctly handles complex spatial reasoning and object interaction tasks that the original model sometimes failed, suggesting that pruning might even act as a regularizer in some cases.
Accuracy vs. Latency trade-offs for different pruning budgets on AI2D and BLINK.

Limitations and Future Work

  • Scheduling Overhead: The current trie implementation's latency is hampered by the overhead of handling many small branches. Future work could focus on better execution schedules that group compatible continuations into larger batches.
  • Architecture Scope: The study focused on the OmniJev architecture (a hybrid of attention and Gated DeltaNet). Extending these findings to a wider variety of multimodal backbones remains an open area.
  • Combined Compression: The authors suggest that combining their layer-wise pruning with token-level pruning (shortening visual sequences) could yield even greater efficiency gains.

Summary updated

Abstract

JEV models make multimodal decisions by directly scoring candidates. Although the common context is encoded once, candidate evaluation can still repeat matching token histories, duplicate inference states, and execute the full backbone. In this paper, we study these sources of redundancy and present FastJEV for compact candidate evaluation. We jointly organize history reuse and state storage, since sharing computation requires preserving states for later branches. We first introduce shared context anchoring to reuse recurrent initial states and omit unused final recurrent caches. We extend this reuse through candidate prefix sharing, retaining the intermediate states needed by subsequent branches. To further reduce the depth of these paths, we apply decision guided pruning based on relative score changes measured on a small unlabeled set. Our method retains full context encoding and all candidates without additional training. We evaluate FastJEV across three OmniJev model sizes on five public benchmarks and reconstructed LIBERO-10 offline questions. At the selected pruning budgets, the complete method reduces candidate depth by 43.75% to 45.83%, while retaining 93.66% to 97.52% of the original task scores on average across the six evaluation sets. Through controlled experiments, we show how candidate overlap and branching structure affect the execution cost of history reuse. In our implementation, candidate prefix sharing can reduce repeated computation while increasing latency. These findings motivate designing sharing granularity and execution schedules together for efficient JEV inference.
BibTeX
@misc{ma2026fastjev,
  author = {Jie Ma and Jie Gao and Yihang Liu and Zhike Qiu and Junle Li and Chongyi Zhuang and Jiayi Ji and Xiaoshuai Sun},
  title = {{FastJEV: Understanding Redundancy for Compact JEV Inference}},
  year = {2026},
  eprint = {2610.11379},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.11379},
}
Figure