Generating temporally coherent high fidelity video is an important milestone in generative modeling research. We make progress towards this milestone by proposing a diffusion model for video generation that shows very promising initial results. Our model is a natural extension of the standard image diffusion architecture, and it enables jointly training from image and video data, which we find to reduce the variance of minibatch gradients and speed up optimization. To generate long and higher resolution videos we introduce a new conditional sampling technique for spatial and temporal video extension that performs better than previously proposed methods. We present the first results on a large text-conditioned video generation task, as well as state-of-the-art results on established benchmarks for video prediction and unconditional video generation. Supplementary material is available at https://video-diffusion.github.io/
核心贡献 · Key contributions
提出一种视频生成的扩散模型，作为图像扩散架构的自然扩展。 Proposes a diffusion model for video generation as a natural extension of image diffusion architectures.
支持图像和视频数据的联合训练，降低小批量梯度方差并加速优化。 Enables joint training on image and video data, reducing minibatch gradient variance and speeding optimization.
引入重建引导采样用于空间和时间视频扩展，优于先前方法。 Introduces reconstruction-guided sampling for spatial and temporal video extension, outperforming prior methods.
在视频预测和无条件视频生成基准上取得最先进结果。 Achieves state-of-the-art results on video prediction and unconditional video generation benchmarks.
首次展示使用扩散模型的大规模文本条件视频生成结果。 Presents first results on large-scale text-conditioned video generation using diffusion models.
展示了无分类器引导和图像-视频联合训练对视频扩散模型的益处。 Demonstrates benefits of classifier-free guidance and joint image-video training for video diffusion models.
局限 · Limitations
模型在固定帧数（如 16 帧）和有限空间分辨率上训练。 Model trained on fixed number of frames (e.g., 16) and limited spatial resolution.
重建引导依赖于条件数据的高斯近似，在 t 较大时可能不准确。 Reconstruction guidance relies on Gaussian approximation of conditioning data, which may be inaccurate for large t.
评估指标（FVD、FID、IS）可能无法完全捕捉时间连贯性或感知质量。 Evaluation metrics (FVD, FID, IS) may not fully capture temporal coherence or perceptual quality.
文本到视频模型继承训练数据的偏差，可能需要筛选以确保公平结果。 Text-to-video models inherit biases from training data and may require curation for fair results.
模型因潜在有害用途未发布，限制了可重复性和进一步研究。 Models not released due to potential for harmful use, limiting reproducibility and further research.
论文章节 · Sections（共 11）
摘要Abstract
1 引言1 Introduction
2 背景2 Background
3 视频扩散模型3 Video diffusion models
3.1 基于重建引导采样的条件生成改进3.1 Reconstruction-guided sampling for improved conditional generation