about
ToonCrafter: Generative Cartoon Interpolation (arxiv.org)
3 points by lmariscal on Jun 2, 2024 | hide | past | pdf | 2 comments on HN

In plain words: ToonCrafter generates the missing frames between two cartoon drawings from scratch, using motion knowledge learned from real videos plus the two end frames and optional sketch guides. It handles cartoons' wild leaps and hidden areas far better than older matching-based tools.

Abstract

We introduce ToonCrafter, a novel approach that transcends traditional correspondence-based cartoon video interpolation, paving the way for generative interpolation. Traditional methods, that implicitly assume linear motion and the absence of complicated phenomena like dis-occlusion, often struggle with the exaggerated non-linear and large motions with occlusion commonly found in cartoons, resulting in implausible or even failed interpolation results. To overcome these limitations, we explore the potential of adapting live-action video priors to better suit cartoon interpolation within a generative framework. ToonCrafter effectively addresses the challenges faced when applying live-action video motion priors to generative cartoon interpolation. First, we design a toon rectification learning strategy that seamlessly adapts live-action video priors to the cartoon domain, resolving the domain gap and content leakage issues. Next, we introduce a dual-reference-based 3D decoder to compensate for lost details due to the highly compressed latent prior spaces, ensuring the preservation of fine details in interpolation results. Finally, we design a flexible sketch encoder that empowers users with interactive control over the interpolation results. Experimental results demonstrate that our proposed method not only produces visually convincing and more natural dynamics, but also effectively handles dis-occlusion. The comparative evaluation demonstrates the notable superiority of our approach over existing competitors.

Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, Tien-Tsin Wong
arXiv:2405.17933 · cs.CV · submitted May 28, 2024
abstract · pdf · html · Project page: https://doubiiu.github.io/projects/ToonCrafter/

add comment on HN

I tried it. Some things work better than others, as you can imagine. Considering it's an opensource I expect a lot from the community. Github repository is already forked a dozen of times (windows fix so far). Also comfyui plugin is on its way