Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling

Chaoqun Cui, Shijing Wang, Liangbin Huang, Qingqing Gu, Zhaolong Huang, Xiao Zeng, Wenji Mao

TL;DR

Hermes is a novel, multimodal large language model (LLM) framework designed specifically for interlingual subtitle translation. By integrating audio-visual speaker diarization, automated terminology distillation, and a novel segment-wise preference optimization method (SAPO), it significantly improves translation coherence, pronoun accuracy, and artistic expressiveness over general-purpose translation models.

What Problem Does This Paper Address?

Translating subtitles for movies and TV shows (interlingual subtitling) is a major bottleneck in entertainment localization. Unlike standard document translation, subtitle text is highly fragmented, deeply tied to audio-visual context, and heavily reliant on colloquial, expressive language.

Current machine translation systems and general-purpose LLMs stumble on four major hurdles unique to subtitles:

  1. Semantic Coherence: LLM context windows require chunking subtitles into arbitrary blocks, which often breaks up continuous dialogue and destroys narrative flow.
  2. Pronoun Reference: Many languages (like Japanese, Korean, or Chinese) frequently drop pronouns or use complex honorific systems. Without knowing who is speaking to whom, translations become inaccurate.
  3. Terminology Consistency: Fictional worlds, fantasy series, and dramas contain specialized terms that must remain consistent across an entire series.
  4. Translation Expressiveness: Standard translation models optimize for literal accuracy, resulting in dry, robotic dialogue. Subtitles require a balance of naturalness and "vividness" to preserve the emotional tone and atmosphere of the original media.

Methods

The authors introduce Hermes, a unified framework that tackles these challenges through three specialized modules:

Figure 1: The overall structure of the unified framework Hermes to enhance expressiveness for interlingual subtitling.

1. Multimodal Speaker Diarization (SD)

To fix pronoun drops and intelligently segment subtitle blocks, Hermes combines video and audio. It uses face detection (TalkNet) and face embeddings to cluster characters visually, treating these high-accuracy visual clusters as "speaker anchors." It then maps audio timbre embeddings (via ERes2NetV2) to these anchors using a voting mechanism. To handle "audio-visual asynchrony"—instances where a character speaks off-screen (roughly 35% of lines in their test set)—Hermes applies a speaker supplementation strategy based on acoustic similarity thresholds between adjacent lines.

Figure 2: Speaker registration.

2. Terminology Identification (TI)

Instead of relying on static dictionaries, Hermes uses a knowledge-distillation approach. A powerful teacher LLM (Qwen-Max) extracts proper nouns and domain-specific terms from bilingual training data. These are filtered via a voting mechanism to build a clean dataset used to train a specialized, smaller terminology LLM ($\pi_{term}$). At inference time, this model extracts and maps consistent terminology directly into the translation prompt.

3. Segment-wise Adaptive Preference Optimization (SAPO)

Standard preference alignment techniques like Direct Preference Optimization (DPO) operate on the entire model response. This is too coarse for subtitles, where a single prompt contains multiple lines of dialogue, leading to "gradient dilution" (a great translation of line 1 masks a terrible translation of line 2).

SAPO introduces a segment-level sampling strategy. An LLM-as-a-Judge evaluates and scores individual line candidates. Hermes then uses a fine-grained alignment loss with an adaptive weighting mechanism. This mechanism includes a gating function that filters out low-diversity, trivial lines (e.g., "Good morning") and focuses optimization purely on lines where expressiveness and localization matter.

Datasets

The authors collected and released a large-scale, multi-directional subtitle dataset sourced from the Youku streaming platform. The dataset spans multiple genres (films, TV series, documentaries, animations) and includes several cross-language family pairs:

  • Directions: en$\Rightarrow$de, en$\Rightarrow$fr, en$\Rightarrow$zh, ko$\Rightarrow$zh, zh$\Rightarrow$en, and zh$\Rightarrow$th.
  • Scale: Each direction contains 100–200 programs, totaling millions of lines of dialogue aligned with their corresponding audio and video tracks.

Key Insights and Ideas

  • Visual Anchors Solve Audio Noise: Pure audio speaker diarization is notoriously noisy in entertainment media due to background music, sound effects, and overlapping dialogue. Fusing vision and audio by using face-clustering as a ground-truth anchor dramatically stabilizes speaker tracking.
  • The "Vividness" Sweet Spot: When training LLMs for expressiveness, more data isn't always better. The authors discovered that as training steps increase, "vividness" scores rise, but literal accuracy eventually drops due to hallucination and over-localization. Finding the exact inflection point (75 steps in their configuration) is vital.
Figure 6: Impact of training data volume on performance.
  • Granular Supervision is Mandatory: Moving from prompt-level rewards to segment-level rewards prevents the model from ignoring subtle contextual errors in multi-turn dialogues.

Key Results

  • Speaker Diarization SOTA: Hermes' SD module outperformed standard baselines (VBx, Spectral Clustering) and the tri-modal SOTA method (E2CP), dropping the Diarization Error Rate (DER) from ~0.13 to ~0.08 on Chinese program test sets.
  • Superior Expressiveness: Evaluated by an LLM-as-a-Judge ensemble (DeepSeek-V3.1, Claude 4, GPT-5) and validated by human blind tests, Hermes consistently achieved the highest "Vividness" scores across all language pairs. It significantly outperformed general-purpose models like GPT-4o and specialized baselines like VideoDubber.
  • Low-Resource Excellence: Hermes showed particularly massive improvements in traditionally difficult, low-resource localization directions like Korean-to-Chinese (ko$\Rightarrow$zh) and Chinese-to-Thai (zh$\Rightarrow$th).

Limitations and Future Work

  • Hallucination Risks: Because SAPO pushes the model toward liberal, creative translations to maximize vividness, it risks introducing hallucinations if over-trained.
  • Computational Overhead: The SAPO sampling phase requires generating $k=15$ candidate translations per line during training, which introduces a heavy computational footprint during the offline alignment phase.
  • Future Directions: The authors suggest exploring tighter, end-to-end multimodal architectures where video and audio features are fed directly into the translation LLM's embedding space, rather than being pre-processed into text-based prompt constraints.

Summary updated

Abstract

Interlingual subtitling, which translates subtitles of visual media into a target language, is essential for entertainment localization but has not yet been explored in machine translation. Although Large Language Models (LLMs) have significantly advanced the general capabilities of machine translation, the distinctive characteristics of subtitle texts pose persistent challenges in interlingual subtitling, particularly regarding semantic coherence, pronoun and terminology translation, and translation expressiveness. To address these issues, we present Hermes, an LLM-based automated subtitling framework. Hermes integrates three modules: Speaker Diarization, Terminology Identification, and Expressiveness Enhancement, which effectively tackle the above challenges. Experiments demonstrate that Hermes achieves state-of-the-art diarization performance and generates expressive, contextually coherent translations, thereby advancing research in interlingual subtitling.
BibTeX
@inproceedings{cui2026hermes,
  author = {Chaoqun Cui and Shijing Wang and Liangbin Huang and Qingqing Gu and Zhaolong Huang and Xiao Zeng and Wenji Mao},
  title = {{Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling}},
  booktitle = {The Web Conference},
  year = {2026},
  doi = {10.1145/3774904.3792262},
}
Figure