STRIVE: Scene Text Replacement In Videos

Vijay Kumar b gOpenReview 2021 · DOI · OpenReview

Abstract

STRIVE (Scene Text Replacement In VidEos) is a novel deep-learning framework designed for realistic text swapping in video sequences. By decoupling style transfer from temporal propagation, STRIVE preserves original style, lighting, and motion while achieving state-of-the-art realism. The system significantly reduces temporal jitter and increases processing speed compared to traditional frame-by-frame editing methods.

Introduction

Replacing text in a single image is a complex task requiring the matching of font, color, texture, and background. In video, these challenges are compounded by temporal factors: changing perspectives, motion blur, and shifting lighting conditions. Previous approaches often relied on frame-by-frame image editing, which typically results in "jitter"—a flickering effect where text appears unstable—and fails to maintain consistent perspective or blur, making the edits appear artificial. STRIVE addresses these issues by treating video text replacement as a disentangled problem of style transfer and temporal propagation.

Methodology

The STRIVE framework utilizes a three-stage modular pipeline to ensure both visual realism and temporal stability:

  1. Pose Normalization (STTN): A Spatio-Temporal Transformer Network (STTN) analyzes text across multiple frames to "frontalize" it. By flattening the text into a standard view, the system simplifies the subsequent editing process.
  2. Reference Frame Replacement (SRNet): The system identifies the highest-quality "reference" frame (typically the sharpest) and performs text replacement using SRNet, a state-of-the-art still-image method. This stage handles the complex task of matching font style and background texture.
  3. Text Propagation Module (TPM): To ensure efficiency and stability, the system propagates the edit from the reference frame to all other frames using learned transforms rather than re-running the expensive SRNet. This module includes a Lighting Correction Module (LCM) to simulate moving light and shadows, and a Blur Prediction Network (BPN) to match the original scene's motion or focus characteristics.

Key Results

STRIVE demonstrates significant improvements over baseline methods like Pix2pix and frame-wise SRNet. Quantitatively, the framework achieved a lower Mean Squared Error (0.0186 vs. 0.0227) and higher OCR accuracy (0.957 vs. 0.771), indicating superior legibility and realism.

Comparison of original video frames (top) and STRIVE-replaced text (bottom) demonstrating temporal consistency.

Crucially, STRIVE reduced temporal jitter by nearly 70% compared to frame-by-frame methods. Efficiency is another major advantage; by performing the primary style transfer only once per sequence, STRIVE is approximately 5x faster than frame-wise alternatives, reaching processing speeds of ~8 frames per second (fps) compared to SRNet’s 1.67 fps.

Conclusion and Future Work

STRIVE represents a significant advancement in video editing by successfully disentangling style and motion. While highly effective for planar surfaces, the current model assumes flat geometry and does not explicitly handle complex occlusions or self-lit text (e.g., neon signs). Future research may extend this approach to 3D non-planar surfaces and more complex environmental interactions.

Summary updated

Abstract

We propose replacing scene text in videos using deep style transfer and learned photometric transformations.Building on recent progress on still image text replacement,we present extensions that alter text while preserving the appearance and motion characteristics of the original video.Compared to the problem of still image text replacement,our method addresses additional challenges introduced by video, namely effects induced by changing lighting, motion blur, diverse variations in camera-object pose over time,and preservation of temporal consistency. We parse the problem into three steps. First, the text in all frames is normalized to a frontal pose using a spatio-temporal trans-former network. Second, the text is replaced in a single reference frame using a state-of-art still-image text replacement method. Finally, the new text is transferred from the reference to remaining frames using a novel learned image transformation network that captures lighting and blur effects in a temporally consistent manner. Results on synthetic and challenging real videos show realistic text trans-fer, competitive quantitative and qualitative performance,and superior inference speed relative to alternatives. We introduce new synthetic and real-world datasets with paired text objects. To the best of our knowledge this is the first attempt at deep video text replacement.
BibTeX
@misc{g2021strive,
  author = {Vijay Kumar b g},
  title = {{STRIVE: Scene Text Replacement In Videos}},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01428},
  howpublished = {OpenReview},
}
Figure