We present CogVideoX, a large-scale text-to-video generation model based on diffusion transformer, which can generate 10-second continuous videos aligned with text prompt, with a frame rate of 16 fps and resolution of 768 * 1360 pixels. Previous video generation models often had limited movement and short durations, and is difficult to generate videos with coherent narratives based on text. We propose several designs to address these issues. First, we propose a 3D Variational Autoencoder (VAE) to compress videos along both spatial and temporal dimensions, to improve both compression rate and video fidelity. Second, to improve the text-video alignment, we propose an expert transformer with the expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. Third, by employing a progressive training and multi-resolution frame pack technique, CogVideoX is adept at producing coherent, long-duration, different shape videos characterized by significant motions. In addition, we develop an effective text-video data processing pipeline that includes various data preprocessing strategies and a video captioning method, greatly contributing to the generation quality and semantic alignment. Results show that CogVideoX demonstrates state-of-the-art performance across both multiple machine metrics and human evaluations. The model weight of both 3D Causal VAE, Video caption model and CogVideoX are publicly available at https://github.com/THUDM/CogVideo.
核心贡献 · Key contributions
提出 CogVideoX，一个基于扩散 Transformer 的大规模文生视频模型，采用 3D 因果 VAE 和专家 Transformer，生成 10 秒 768×1360 分辨率 16fps 的视频。 Proposes CogVideoX, a large-scale text-to-video diffusion Transformer with 3D causal VAE and expert Transformer, generating 10s 768x1360 videos at 16fps.
引入 3D 因果 VAE，在空间和时间维度压缩视频，提高压缩率和视频保真度，减少闪烁。 Introduces a 3D causal VAE that compresses videos spatially and temporally, improving compression rate and video fidelity while reducing flicker.
设计具有专家自适应层归一化的专家 Transformer，实现文本与视频模态深度融合，增强文本-视频对齐。 Designs an expert Transformer with expert adaptive LayerNorm for deep fusion of text and video modalities, enhancing text-video alignment.
采用渐进式训练和多分辨率帧打包技术，生成连贯、长时长、形状各异且运动显著的视频。 Employs progressive training and multi-resolution frame packing to generate coherent, long-duration videos with various shapes and significant motions.
构建有效的数据管道，包括视频过滤和视频字幕生成，大幅提升生成质量和语义对齐。 Builds an effective data pipeline including video filtering and video captioning, greatly boosting generation quality and semantic alignment.
开源 5B 和 2B 模型（文生视频和图生视频），首个商用级开源视频生成模型。 Open-sources 5B and 2B models (text-to-video and image-to-video), the first commercial-grade open-source video generation models.
局限 · Limitations
最大视频长度仅 10 秒，对复杂叙事仍有限；生成更长时间一致运动视频仍具挑战。 Maximum video length is 10 seconds, still limited for complex narratives; generating longer videos with consistent motion remains challenging.
物理模拟（如物体交互、流体动力学）不完美；生成视频可能违反真实物理规律。 Physics simulation (e.g., object interaction, fluid dynamics) is imperfect; generated videos may violate real-world physics.
训练数据虽经过滤但仍包含噪声网络视频；残留低质量数据可能影响生成质量。 Training data contains noisy web videos despite filtering; residual low-quality data may affect generation quality.
复杂长文本指令跟随能力有限；模型难以处理细节丰富的叙事描述。 Complex long-text instruction following is limited; the model struggles with detailed narrative descriptions.
推理计算和内存开销大，限制了在消费级硬件上的部署。 Inference is computationally expensive and memory-intensive, limiting deployment on consumer hardware.
论文章节 · Sections（共 14）
摘要Abstract
1 引言1 Introduction
2 CogVideoX 架构2 The CogVideoX Architecture
2.2 专家 Transformer2.2 Expert Transformer
3 训练 CogVideoX3 Training CogVideoX
3.1 多分辨率帧打包3.1 Multi-Resolution Frame Pack
3.2 渐进式训练3.2 Progressive Training
3.3 显式均匀采样3.3 Explicit Uniform Sampling
3.4 数据3.4 Data
4.1 消融实验4.1 Ablation Study
4.2 评估4.2 Evaluation
5 结论5 Conclusion
3. 元素不完整 4. 特征不准确3 Elements are incomplete 4. Features are inaccurate
K 数据过滤细节0 Cover image resolution is low, image is blurry.