We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.
核心贡献 · Key contributions
首个从互联网视频中无监督训练的生成式交互环境。 First generative interactive environment trained unsupervised from Internet videos.
11B 参数的基础世界模型，包含时空视频分词器、动力学模型和潜在动作模型。 11B parameter foundation world model with spatiotemporal video tokenizer, dynamics model, and latent action model.
通过学习的潜在动作空间实现无动作标签的逐帧控制。 Enables frame-by-frame control without action labels via learned latent action space.
规模分析表明性能随模型和批次大小提升。 Scaling analysis shows performance improves with model and batch size.
互联网视频的潜在动作可训练智能体在未见环境中模仿行为。 Latent actions from Internet videos can train agents to imitate behaviors in unseen environments.
泛化到文本、草图、照片等多种提示。 Generalizes to diverse prompts including text, sketches, and photos.
局限 · Limitations
限于 16 帧记忆，导致长时程环境不一致。 Limited to 16-frame memory, causing inconsistent environments over long horizons.
运行速度约 1 FPS，需未来改进以实现交互帧率。 Operates at ~1 FPS, requiring future advances for interactive frame rates.
可能像其他自回归模型一样产生不现实的未来幻觉。 May hallucinate unrealistic futures like other autoregressive models.
仅在 2D 平台游戏和机器人视频上训练，限制了领域通用性。 Trained only on 2D platformer and robotics videos, limiting domain generality.
论文章节 · Sections（共 18）
摘要Abstract
1 引言1 1. Introduction
2 方法2 2. Methodology
2.1 模型组件2.1 Model Components
2.2 推理：动作可控视频生成2.2 Inference: Action-Controllable Video Generation