about
Tunnel Try-On: Excavating Spatial-Temporal Tunnels for Virtual Try-On in Videos (arxiv.org)
3 points by Jimmc414 on Apr 29, 2024 | hide | past | pdf | discuss on HN

In plain words: It zooms in on the clothing area of each frame, using smoothed crops and outside context so the fabric's fine details survive while the person moves. Unlike earlier video try-on tools that blur the garment or jitter, it keeps details sharp and motion steady.

Abstract · Tunnel Try-on: Excavating Spatial-temporal Tunnels for High-quality Virtual Try-on in Videos

Video try-on is a challenging task and has not been well tackled in previous works. The main obstacle lies in preserving the details of the clothing and modeling the coherent motions simultaneously. Faced with those difficulties, we address video try-on by proposing a diffusion-based framework named "Tunnel Try-on." The core idea is excavating a "focus tunnel" in the input video that gives close-up shots around the clothing regions. We zoom in on the region in the tunnel to better preserve the fine details of the clothing. To generate coherent motions, we first leverage the Kalman filter to construct smooth crops in the focus tunnel and inject the position embedding of the tunnel into attention layers to improve the continuity of the generated videos. In addition, we develop an environment encoder to extract the context information outside the tunnels as supplementary cues. Equipped with these techniques, Tunnel Try-on keeps the fine details of the clothing and synthesizes stable and smooth videos. Demonstrating significant advancements, Tunnel Try-on could be regarded as the first attempt toward the commercial-level application of virtual try-on in videos.

Zhengze Xu, Mengting Chen, Zhao Wang, Linyu Xing, Zhonghua Zhai, Nong Sang, Jinsong Lan, Shuai Xiao, Changxin Gao
arXiv:2404.17571 · cs.CV · submitted Apr 26, 2024
abstract · pdf · html · Project Page: https://mengtingchen.github.io/tunnel-try-on-page/

add comment on HN