TL;DR
Omni-JEV is a unified model for full-duplex voice assistants that makes interaction decisions—like whether to start, stop, or continue speaking—based on both text and dual-stream audio. By conditioning these decisions on the assistant's current state and conversation history, it achieves high accuracy (84.08% for audio, 96.55% for text) in complex scenarios where user intent alone is ambiguous.
What Problem Does This Paper Address?
In a "full-duplex" conversation, a voice assistant listens while it is speaking. This creates a significant challenge: a user's utterance (like "Mm-hm" or "Wait") doesn't have a single fixed meaning. Its impact depends entirely on what the assistant is doing at that moment.
If the assistant is idle, "Mm-hm" might just be a backchannel to ignore. If the assistant is currently speaking, it could be an acknowledgment to continue or a precursor to an interruption. Current models often struggle with this ambiguity because they lack a unified "action policy" that accounts for the assistant's internal state (e.g., playing, interrupted, or idle). Omni-JEV addresses this by formulating interaction control as a state-conditioned classification task across five core actions: listen, start, continue, stop, and recover.
Methods
Omni-JEV is built on the MiniMind-O framework, utilizing a Qwen3-based "Thinker" as its core language model. Key technical innovations include:
- Dual-Stream Audio Encoding: Unlike standard models that mix all audio into one stream, Omni-JEV uses a shared acoustic frontend to encode user speech and assistant-reference audio (the assistant's own playback) into separate "role spans." This allows the model to clearly distinguish between the user's voice and its own echo.
- Unified Sequence Modeling: The model processes a single serialized prompt containing instructions, role-indexed audio/text, interaction context (like the current state), and candidate action descriptions. This allows the same "Thinker" to handle both text and speech inputs seamlessly.
- RLCD (Reinforcement Learning for Probability-Sensitive Decisions): After initial supervised fine-tuning (SFT), the authors apply RLCD to refine the model's output probabilities. They compare two strategies: Gaussian-logit exploration (perturbing logits with noise) and categorical Choice (sampling discrete candidates). This step is crucial for a controller that needs to know not just what action to take, but how certain it should be.
Datasets
The authors constructed a large-scale bilingual (English and Chinese) weak-label corpus by aggregating and re-labeling nine existing source families, including:
- Conversational data: Intelligent Wakeup, SmoothConv, and synthetic full-duplex datasets (FD_data v2/v3).
- Command and Intent data: BANKING77, MASSIVE, and MInDS-14.
- Environmental sounds: ESC-10.
- Keyword spotting: Mini Speech Commands.
The final corpus contains approximately 313,000 training instances and 35,000 test instances, providing a diverse range of interaction scenarios.
Key Insights and Ideas
- State Matters More Than Content: The core "aha moment" is that interaction control is not just about understanding what was said, but when it was said relative to the assistant's state.
- The Power of Dual Streams: By keeping the assistant's playback as a separate reference stream, the model can effectively handle overlaps and distinguish substantive interruptions from simple acknowledgments.
- RLCD Trade-offs: The comparison between Gaussian and categorical RLCD revealed a fascinating trade-off: Gaussian exploration tends to yield better classification balance (higher Macro-F1), while categorical sampling produces more reliable probability distributions (lower Brier error).
- Cross-Modal Consistency: Using a shared backbone for text and speech allows the model to leverage high-quality text-based reasoning to improve its performance on noisier audio inputs.
Key Results
- High Accuracy: The Gaussian-tuned Omni-JEV achieved 84.08% accuracy on audio and 96.55% on text, outperforming several external full-scope systems (like Qwen3-Omni and MiniCPM-o) that were tested zero-shot on the same task.
- Robust Classification: It reached an audio Macro-F1 of 76.00%, indicating it performs well even on rare actions like
recover. - Probability Quality: Categorical RLCD significantly reduced probability errors, achieving a Brier score roughly 11% lower than the Gaussian version on audio tasks.
Limitations and Future Work
- Weak Supervision: The targets are "weak" labels derived from rules and other LLMs, which may not perfectly capture the nuances of human-to-human interaction.
- Real-time Metrics: While the model performs well on collected segments, the paper does not measure real-time factors like end-to-end latency or robustness to varying acoustic echo conditions in a live loop.
- Data Imbalance: Some actions, particularly
recover(resuming after an interruption), are extremely rare in the dataset, making them harder to learn and evaluate reliably. - Future Directions: The authors suggest exploring multi-turn utility, improving acoustic robustness, and integrating the model into a closed-loop interactive controller.
Summary updated