We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. Recently, latent diffusion models trained for 2D image synthesis have been turned into generative video models by inserting temporal layers and finetuning them on small, high-quality video datasets. However, training methods in the literature vary widely, and the field has yet to agree on a unified strategy for curating video data. In this paper, we identify and evaluate three different stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning. Furthermore, we demonstrate the necessity of a well-curated pretraining dataset for generating high-quality videos and present a systematic curation process to train a strong base model, including captioning and filtering strategies. We then explore the impact of finetuning our base model on high-quality data and train a text-to-video model that is competitive with closed-source video generation. We also show that our base model provides a powerful motion representation for downstream tasks such as image-to-video generation and adaptability to camera motion-specific LoRA modules. Finally, we demonstrate that our model provides a strong multi-view 3D-prior and can serve as a base to finetune a multi-view diffusion model that jointly generates multiple views of objects in a feedforward fashion, outperforming image-based methods at a fraction of their compute budget. We release code and model weights at https://github.com/Stability-AI/generative-models .
核心贡献 · Key contributions
提出系统性的数据筛选流程，将大规模未整理视频集合转化为适用于生成式视频建模的高质量数据集。 Proposes a systematic data curation workflow to transform large uncurated video collections into quality datasets for generative video modeling.
识别并评估视频潜在扩散模型的三个不同训练阶段：文本到图像预训练、视频预训练和高品质视频微调。 Identifies and evaluates three distinct training stages for video latent diffusion models: text-to-image pretraining, video pretraining, and high-quality video finetuning.
训练出在人类偏好研究中优于先前模型的文本到视频和图像到视频模型。 Trains state-of-the-art text-to-video and image-to-video models outperforming prior models in human preference studies.
展示基础模型为图像到视频和相机运动 LoRA 模块等下游任务提供强大的运动表示。 Demonstrates that the base model provides a powerful motion representation for downstream tasks like image-to-video and camera motion LoRA modules.
表明视频扩散模型可微调为强大的多视角生成器，以更少算力超越专用方法。 Shows that the video diffusion model can be finetuned into a strong multi-view generator, outperforming specialized methods with less compute.
公开代码和模型权重以促进可重复性和进一步研究。 Releases code and model weights to facilitate reproducibility and further research.
局限 · Limitations
模型仅能生成最多 25 帧的视频，限制了长时长内容。 The model is limited to generating videos of up to 25 frames, restricting long-duration content.
性能高度依赖预训练数据集的质量和筛选，可能无法泛化到所有领域。 Performance depends heavily on the quality and curation of the pretraining dataset, which may not generalize to all domains.
研究聚焦于潜在扩散模型，可能不适用于其他视频生成架构。 The study focuses on latent diffusion models and may not apply to other video generation architectures.
人类偏好评估具有主观性，可能无法涵盖视频质量的所有方面。 Human preference evaluations are subjective and may not capture all aspects of video quality.
多视角微调限于以物体为中心的数据集，可能无法泛化到复杂场景。 The multi-view finetuning is limited to object-centric datasets and may not generalize to complex scenes.
论文章节 · Sections（共 16）
摘要Abstract
1 引言1 Introduction
2 背景2 Background
3 为高质量视频合成整理数据3 Curating Data for HQ Video Synthesis
3.1 数据处理与标注3.1 Data Processing and Annotation
3.2 阶段一：图像预训练3.2 Stage I: Image Pretraining
3.3 阶段二：整理视频预训练数据集3.3 Stage II: Curating a Video Pretraining Dataset