Omni-JEV: Text and Dual-Stream Speech Decisions

Shengzhou Kong

TL;DR

Omni-JEV is a unified model for full-duplex voice assistants that makes interaction decisions—like whether to start, stop, or continue speaking—based on both text and dual-stream audio. By conditioning these decisions on the assistant's current state and conversation history, it achieves high accuracy (84.08% for audio, 96.55% for text) in complex scenarios where user intent alone is ambiguous.

What Problem Does This Paper Address?

In a "full-duplex" conversation, a voice assistant listens while it is speaking. This creates a significant challenge: a user's utterance (like "Mm-hm" or "Wait") doesn't have a single fixed meaning. Its impact depends entirely on what the assistant is doing at that moment.

If the assistant is idle, "Mm-hm" might just be a backchannel to ignore. If the assistant is currently speaking, it could be an acknowledgment to continue or a precursor to an interruption. Current models often struggle with this ambiguity because they lack a unified "action policy" that accounts for the assistant's internal state (e.g., playing, interrupted, or idle). Omni-JEV addresses this by formulating interaction control as a state-conditioned classification task across five core actions: listen, start, continue, stop, and recover.

Methods

Omni-JEV is built on the MiniMind-O framework, utilizing a Qwen3-based "Thinker" as its core language model. Key technical innovations include:

  • Dual-Stream Audio Encoding: Unlike standard models that mix all audio into one stream, Omni-JEV uses a shared acoustic frontend to encode user speech and assistant-reference audio (the assistant's own playback) into separate "role spans." This allows the model to clearly distinguish between the user's voice and its own echo.
  • Unified Sequence Modeling: The model processes a single serialized prompt containing instructions, role-indexed audio/text, interaction context (like the current state), and candidate action descriptions. This allows the same "Thinker" to handle both text and speech inputs seamlessly.
  • RLCD (Reinforcement Learning for Probability-Sensitive Decisions): After initial supervised fine-tuning (SFT), the authors apply RLCD to refine the model's output probabilities. They compare two strategies: Gaussian-logit exploration (perturbing logits with noise) and categorical Choice (sampling discrete candidates). This step is crucial for a controller that needs to know not just what action to take, but how certain it should be.
The Omni-JEV architecture independently encodes user and assistant audio before processing them through a shared Transformer backbone.

Datasets

The authors constructed a large-scale bilingual (English and Chinese) weak-label corpus by aggregating and re-labeling nine existing source families, including:

  • Conversational data: Intelligent Wakeup, SmoothConv, and synthetic full-duplex datasets (FD_data v2/v3).
  • Command and Intent data: BANKING77, MASSIVE, and MInDS-14.
  • Environmental sounds: ESC-10.
  • Keyword spotting: Mini Speech Commands.

The final corpus contains approximately 313,000 training instances and 35,000 test instances, providing a diverse range of interaction scenarios.

Key Insights and Ideas

  • State Matters More Than Content: The core "aha moment" is that interaction control is not just about understanding what was said, but when it was said relative to the assistant's state.
  • The Power of Dual Streams: By keeping the assistant's playback as a separate reference stream, the model can effectively handle overlaps and distinguish substantive interruptions from simple acknowledgments.
  • RLCD Trade-offs: The comparison between Gaussian and categorical RLCD revealed a fascinating trade-off: Gaussian exploration tends to yield better classification balance (higher Macro-F1), while categorical sampling produces more reliable probability distributions (lower Brier error).
  • Cross-Modal Consistency: Using a shared backbone for text and speech allows the model to leverage high-quality text-based reasoning to improve its performance on noisier audio inputs.
Interaction decisions depend on both the user's intent and the assistant's current playback state.

Key Results

  • High Accuracy: The Gaussian-tuned Omni-JEV achieved 84.08% accuracy on audio and 96.55% on text, outperforming several external full-scope systems (like Qwen3-Omni and MiniCPM-o) that were tested zero-shot on the same task.
  • Robust Classification: It reached an audio Macro-F1 of 76.00%, indicating it performs well even on rare actions like recover.
  • Probability Quality: Categorical RLCD significantly reduced probability errors, achieving a Brier score roughly 11% lower than the Gaussian version on audio tasks.

Limitations and Future Work

  • Weak Supervision: The targets are "weak" labels derived from rules and other LLMs, which may not perfectly capture the nuances of human-to-human interaction.
  • Real-time Metrics: While the model performs well on collected segments, the paper does not measure real-time factors like end-to-end latency or robustness to varying acoustic echo conditions in a live loop.
  • Data Imbalance: Some actions, particularly recover (resuming after an interruption), are extremely rare in the dataset, making them harder to learn and evaluate reliably.
  • Future Directions: The authors suggest exploring multi-turn utility, improving acoustic robustness, and integrating the model into a closed-loop interactive controller.

Summary updated

Abstract

A user utterance does not uniquely determine the next action in full-duplex dialogue: its consequences depend on the assistant's response state and interaction history. Speech activity, utterance completion, and interruption intent provide related evidence, whose implications for a response require an explicit action policy. We formulate this task at collected-segment endpoints and introduce Omni-JEV, which maps text or role-indexed user and assistant-reference speech to probabilities over five interaction actions. An aligned speech-to-speech understanding backbone encodes the acoustic roles separately and shares its candidate readout with text inputs. Bilingual weak supervision supports cross-entropy fine-tuning followed by reinforcement learning for probability-sensitive decisions (RLCD). We compare Gaussian-logit and categorical Choice RLCD from the same supervised initialization under matched training exposure, observing different trade-offs between classification and probability quality. Gaussian Omni-JEV achieves test accuracies of 84.08% for audio and 96.55% for text. This common action interface supports decisions about when a full-duplex voice assistant should respond, preserve ongoing speech, interrupt, or resume an interrupted response.
BibTeX
@misc{kong2026omni,
  author = {Shengzhou Kong},
  title = {{Omni-JEV: Text and Dual-Stream Speech Decisions}},
  year = {2026},
  eprint = {2611.13777},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13777},
}
Figure