Qwen-UI-Agent 技术报告:迈向下一代以真实世界为中心的基座 GUI 智能体

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

杨安 An Yang · · 2026-07-30 · arXiv:2607.28227 ↗

打开互动全文版(逐段中英对照 + 图/公式 + 论文问答)→

摘要 · Abstract

GUI 智能体有潜力成为现有数字设备上的通用执行器。为了推动其走向真实世界应用,我们设想智能体能够在真实设备上可靠运行、跨平台执行工作流、将 GUI 交互与 CLI 执行相结合、完成长周期任务、主动发起有用服务,并在最小人工干预下自主提升能力。在这一愿景的指引下,我们提出了 Qwen-UI-Agent,一个以真实世界为中心的基座 GUI 智能体,覆盖移动设备、电脑使用、网页和 DeepSearch 环境。Qwen-UI-Agent 将多样化的沙盒环境与大规模真实设备移动运行时相结合。其统一动作空间将 GUI 操作与 CLI 执行交错进行,并在单次模型回合中生成批量动作。一个 AutoResearch 风格的数据飞轮利用智能体来构建任务和环境、诊断失败并规划后续迭代。在线强化学习支持在超过 100 回合的轨迹上进行训练,超过 10,000 个并发环境加速了回放。一个轻量级 harness 层支持跨移动设备和电脑的主动服务启动和有状态工作流。在广泛的评估套件中,Qwen-UI-Agent 在移动使用基准上取得了最先进的性能,同时在计算机和浏览器使用任务中与前沿模型(包括 Opus 4.8、Gemini 3.1 Pro 和 GPT-5.6 Sol)相比具有竞争力的表现。在移动使用方面,它在 MobileWorld 上达到 82.1%,在 MobileWorld-Real 上达到 92.2%,在 AndroidDaily 上达到 97.5%。在计算机使用方面,它在 OSWorld-Verified 上达到 79.5%,在 OSWorld-v2 上获得 40.0% 的部分进度得分。在浏览器使用和 GUI 定位方面,它分别在 WebArena 上达到 73.6%,在 ScreenSpot-Pro 上达到 81.5%。

GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.

核心贡献 · Key contributions

局限 · Limitations

论文章节 · Sections(共 39)

阅读逐段中英对照全文 →