Generating Consistent Long Depth Sequences for Open-world Videos

arXiv 2024

Wenbo Hu^{1* †} Xiangjun Gao^2* Xiaoyu Li^{1* †} Sijie Zhao¹ Xiaodong Cun¹
Yong Zhang¹ Long Quan² Ying Shan^{3, 1}

¹Tencent AI Lab ²The Hong Kong University of Science and Technology ³ARC Lab, Tencent PCG

Paper arXiv Code 🤗 Hugging Face Demo

We innovate DepthCrafter, a novel video depth estimation approach, by leveraging video diffusion models. It can generate temporally consistent long depth sequences with fine-grained details for open-world videos, without requiring additional information such as camera poses or optical flow.

Brief introduction

Motivation. Despite significant advancements in monocular depth estimation for static images, estimating video depth in the open world remains challenging, since open-world videos are extremely diverse in content, motion, camera movement, and length. We present DepthCrafter, an innovative method for generating temporally consistent long depth sequences with intricate details for open-world videos, without requiring any supplementary information such as camera poses or optical flow. DepthCrafter achieves generalization ability to open-world videos by training a video-to-depth model from a pre-trained image-to-video diffusion model, through our meticulously designed three-stage training strategy with the compiled paired video-depth datasets. Our training approach enables the model to generate depth sequences with variable lengths at one time, up to 110 frames, and harvest both precise depth details and rich content diversity from realistic and synthetic datasets. We also propose an inference strategy that processes extremely long videos through segment-wise estimation and seamless stitching.

Overview. DepthCrafter is a conditional diffusion model that models the distribution over the depth sequence conditioned on the input video. We train the model in three stages, where the spatial or temporal layers of the diffusion model are progressively learned on our compiled realistic or synthetic datasets with variable lengths. During inference, given an open-world video, it can generate temporally consistent long depth sequences with fine-grained details for the entire video from initialized Gaussian noise, without requiring any supplementary information, such as camera poses or optical flow.

Inference for extremely long videos. We divide the video into overlapped segments and estimate the depth sequences for each segment with a noise initialization strategy to anchor the scale and shift of depth distributions. These estimated segments are then seamlessly stitched together with a latent interpolation strategy to form the entire depth sequence.

Comparison with representative methods on labeled datasets

Comparison with Depth-Anything-V2 on open-world videos

Novel view rendering

Visual effects

BibTeX

@article{hu2024-DepthCrafter,
            author      = {Hu, Wenbo and Gao, Xiangjun and Li, Xiaoyu and Zhao, Sijie and Cun, Xiaodong and Zhang, Yong and Quan, Long and Shan, Ying},
            title       = {DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos},
            journal     = {arXiv preprint arXiv:2409.02095},
            year        = {2024}
    }