自主 UI 智能体由 AI 驱动，有巨大潜力通过自动化日常任务如报税和缴费来提高人类生产力。然而，发挥其全部潜力的一个主要挑战是安全性，而智能体代表用户采取行动的能力加剧了这一挑战。现有的针对网络智能体的提示注入测试要么过于简化威胁，测试不现实的场景或给攻击者过多权力，要么只关注单步孤立任务。为了更准确地衡量安全网络智能体的进展，我们引入了 WASP——一个针对提示注入攻击的端到端评估的新的公开基准。使用 WASP 评估表明，即使顶级 AI 模型，包括具有高级推理能力的模型，也能被非常现实场景中简单、低投入的人为编写的注入所欺骗。我们的端到端评估揭示了一个以前未观察到的洞察：虽然攻击在高达 86%的情况下部分成功，但即使是最先进的智能体也常常难以完全完成攻击者的目标——这突显了当前基于无能的安全状态。
Autonomous UI agents powered by AI have tremendous potential to boost human productivity by automating routine tasks such as filing taxes and paying bills. However, a major challenge in unlocking their full potential is security, which is exacerbated by the agent's ability to take action on their user's behalf. Existing tests for prompt injections in web agents either over-simplify the threat by testing unrealistic scenarios or giving the attacker too much power, or look at single-step isolated tasks. To more accurately measure progress for secure web agents, we introduce WASP -- a new publicly available benchmark for end-to-end evaluation of Web Agent Security against Prompt injection attacks. Evaluating with WASP shows that even top-tier AI models, including those with advanced reasoning capabilities, can be deceived by simple, low-effort human-written injections in very realistic scenarios. Our end-to-end evaluation reveals a previously unobserved insight: while attacks partially succeed in up to 86% of the case, even state-of-the-art agents often struggle to fully complete the attacker goals -- highlighting the current state of security by incompetence.
核心贡献 · Key contributions
提出 WASP，一个用于端到端评估 Web 智能体对抗提示注入攻击安全性的新公开基准。 Proposes WASP, a new publicly available benchmark for end-to-end evaluation of web agent security against prompt injection attacks.
真实的威胁模型：攻击者仅控制网站部分内容而无法完全操控，反映现实场景。 Realistic threat model where attackers control only parts of websites without full access, reflecting real-world scenarios.
端到端评估显示，简单的人工编写的注入在高达 86%的案例中欺骗了顶级 AI 模型。 End-to-end evaluation reveals that simple human-written injections deceive top-tier AI models in up to 86% of cases.
最先进智能体完整完成攻击者目标成功率仅 16%，凸显“无能即安全”现状。 State-of-the-art agents achieve only up to 16% full attacker goal success, highlighting security by incompetence.
基准兼容任何通用 Web 智能体并开源，确保可复现性。 Benchmark compatible with any generalist web agent and open-sourced for reproducibility.
提供中间和端到端攻击成功率，实现细粒度安全评估。 Provides both intermediate and end-to-end attack success rates, offering granular security assessment.
局限 · Limitations
仅支持两个 Web 环境（GitLab 和 Reddit），多样性受限。 Only supports two web environments (GitLab and Reddit), limiting diversity.
缺少多样化的提示注入攻击模板；手工模板可能无法涵盖所有真实变体。 Lacks diverse prompt injection attack prompts; manual templates may not cover all real-world variations.
未报告实验结果的误差条或统计显著性。 No error bars or statistical significance reported for experimental results.
未充分披露计算资源和执行时间细节，影响可复现性。 Insufficient details on compute resources and execution time for reproducibility.
效用测试集（37 个任务）可能无法完全代表真实良性用户场景。 Utility test set (37 tasks) may not fully represent real-world benign user scenarios.
论文章节 · Sections（共 16）
摘要Abstract
1 主张1 Claims
2 局限性2 Limitations
3 理论假设与证明3 Theory assumptions and proofs
4 实验结果可复现性4 Experimental result reproducibility
5 数据和代码的开放获取5 Open access to data and code
6 实验设置/细节6 Experimental setting/details
7 实验统计显著性7 Experiment statistical significance
8 实验计算资源8 Experiments compute resources
9 伦理准则9 Code of ethics
10 更广泛的影响10 Broader impacts
11 安全措施11 Safeguards
现有资产的许可证12 Licenses for existing assets
13 新资产13 New assets
14 众包与人类受试者研究14 Crowdsourcing and research with human subjects