Digital agents for automating tasks across different platforms by directly manipulating the GUIs are increasingly important. For these agents, grounding from language instructions to target elements remains a significant challenge due to reliance on HTML or AXTree inputs. In this paper, we introduce Aria-UI, a large multimodal model specifically designed for GUI grounding. Aria-UI adopts a pure-vision approach, eschewing reliance on auxiliary inputs. To adapt to heterogeneous planning instructions, we propose a scalable data pipeline that synthesizes diverse and high-quality instruction samples for grounding. To handle dynamic contexts in task performing, Aria-UI incorporates textual and text-image interleaved action histories, enabling robust context-aware reasoning for grounding. Aria-UI sets new state-of-the-art results across offline and online agent benchmarks, outperforming both vision-only and AXTree-reliant baselines. We release all training data and model checkpoints to foster further research at https://ariaui.github.io.
核心贡献 · Key contributions
提出 Aria-UI，一种纯视觉的大多模态模型，用于 GUI 定位，不依赖 HTML 或 AXTree 输入。 Proposes Aria-UI, a pure-vision LMM for GUI grounding without relying on HTML or AXTree inputs.
设计可扩展的数据管道，合成多样、高质量的指令样本用于定位。 Designs a scalable data pipeline to synthesize diverse, high-quality instruction samples for grounding.
融入文本和文本-图像交错动作历史，实现鲁棒的上下文感知定位。 Incorporates textual and text-image interleaved action histories for robust context-aware grounding.
在离线和在线智能体基准上取得最先进结果，优于纯视觉和依赖 AXTree 的基线。 Achieves state-of-the-art results on offline and online agent benchmarks, outperforming vision-only and AXTree-reliant baselines.
发布所有训练数据和模型检查点，以促进进一步研究。 Releases all training data and model checkpoints to foster further research.
在网页、桌面和移动平台上展示出强大的零样本泛化能力。 Demonstrates strong zero-shot generalization across web, desktop, and mobile platforms.
局限 · Limitations
缺乏自主规划能力；依赖独立的规划器模型生成任务指令。 Lacks autonomous planning capability; relies on a separate planner model for task instructions.
训练未包含对规划器生成指令的错误纠正。 Training does not include error correction for planner-generated instructions.
当规划器生成低质量或错误指令时，性能可能下降。 Performance may degrade when planner produces low-quality or incorrect instructions.
仅限于 GUI 定位；未涉及智能体任务的其他方面，如规划或记忆。 Limited to GUI grounding; does not address other aspects of agent tasks like planning or memory.
数据管道依赖强大多模态模型（如 GPT-4o）生成描述，可能引入偏差。 Data pipeline relies on strong LMMs (e.g., GPT-4o) for caption generation, which may introduce biases.
论文章节 · Sections（共 14）
摘要Abstract
1 引言1 Introduction
2 方法2 Method
2.1 大规模多样化 GUI 数据合成2.1 Large-scale Diverse GUI Data Synthesizing