TL;DR
FastJEV optimizes Joint Evaluation (JEV) models—which score multiple decision candidates directly—by eliminating redundant computation through shared state anchoring, prefix sharing, and decision-guided pruning. It reduces candidate execution depth by approximately 45% while retaining over 93% of original task performance without requiring any additional training.
What Problem Does This Paper Address?
Multimodal models are increasingly used for embodied AI and decision-making tasks where they must choose the best action from a set of candidates. Unlike standard generative models that produce text token-by-token, Joint Evaluation (JEV) models score a set of explicit alternatives (e.g., "pick up the red mug" vs. "pick up the blue bottle") based on a shared visual and textual context.
However, current JEV inference is inefficient. Even if the context is encoded once, evaluating $N$ different candidates often involves repeating matching token histories, duplicating inference states, and running the entire model backbone for every single choice. This redundancy leads to high latency and memory usage, especially as the number of candidates or the model size grows.
Methods
The authors propose FastJEV, a framework that organizes candidate evaluation into a compact structure using three key components:
- Shared Context Anchoring (S): Instead of copying the recurrent states of the shared context for every candidate branch, FastJEV uses a single read-only buffer. It also omits the allocation of final recurrent caches for "leaf" nodes that don't have further branches, saving memory.
- Candidate Prefix Sharing (T): Candidates often share long prefixes (e.g., "The robot is moving toward the..."). FastJEV organizes these into a trie (prefix tree), ensuring that shared token sequences are processed by the language backbone only once.
- Decision Guided Pruning (P): Not all layers contribute equally to the final decision. The authors use a small, unlabeled calibration set to measure "decision sensitivity"—how much skipping a layer changes the relative ranking of candidates. They then skip the least sensitive layers during candidate evaluation while keeping the full backbone for the initial context encoding.
Datasets
FastJEV was evaluated across three sizes of the OmniJev model (0.8B, 2B, and 4B parameters) using:
- Public Benchmarks: AI2D (diagrams), ScienceQA-IMG (science questions), POPE (object hallucination), BLINK (visual perception), and ERQA (embodied reasoning).
- LIBERO-10: A reconstructed offline question set based on robot manipulation demonstrations to test embodied decision-making.
- Calibration: A small subset of OK-VQA was used for the offline pruning sensitivity assessment.
Key Insights and Ideas
- The Trie Paradox: Interestingly, while candidate prefix sharing (T) significantly reduces the number of logical tokens processed, it can sometimes increase latency. This is because frequent branching creates many small segments, leading to overhead from multiple kernel launches and complex state management.
- Non-Uniform Sensitivity: Pruning sensitivity does not decrease linearly with depth. As shown in Figure 2, the middle-to-late layers often show the lowest sensitivity, while the very final layer is critical for maintaining decision quality.
- Ranking vs. Accuracy: The core insight of decision-guided pruning is that we don't need to preserve the exact logit values, only the relative order of candidates. This allows for much more aggressive pruning than standard task-agnostic methods.
Key Results
- Efficiency: FastJEV reduces candidate execution depth by 43.75% to 45.83%.
- Performance Retention: Across six evaluation sets, the method retains 93.66% to 97.52% of the original model's accuracy.
- Speed and Memory: The "S+P" configuration (Anchoring + Pruning) achieved speedups of 1.05x to 1.26x on an NVIDIA L20 GPU. On the LIBERO robot benchmark, peak memory usage was reduced by up to 15.7%.
- Qualitative Success: The authors demonstrate that even with significant pruning, the model correctly handles complex spatial reasoning and object interaction tasks that the original model sometimes failed, suggesting that pruning might even act as a regularizer in some cases.
Limitations and Future Work
- Scheduling Overhead: The current trie implementation's latency is hampered by the overhead of handling many small branches. Future work could focus on better execution schedules that group compatible continuations into larger batches.
- Architecture Scope: The study focused on the OmniJev architecture (a hybrid of attention and Gated DeltaNet). Extending these findings to a wider variety of multimodal backbones remains an open area.
- Combined Compression: The authors suggest that combining their layer-wise pruning with token-level pruning (shortening visual sequences) could yield even greater efficiency gains.
Summary updated