How well do agents use test/verification techniques?

AI Digest

文章评估AI代理使用测试技术的效果，发现其测试能力不足，即使有指导也难以提升，揭示当前代理在测试方法应用上的局限性。

The study evaluates AI agents' use of testing techniques, finding their testing capabilities are limited even with guidance, highlighting gaps in agentic testing methods.

Key points

• AI代理在测试技术应用上普遍表现不佳，多数方法效果有限 AI agents show limited effectiveness in applying testing techniques despite guidance
• 形式化方法和TDD等技术未达预期效果，代理常机械执行而非深度应用 Formal methods and TDD underperform, with agents often executing superficially
• 测试技能对代理表现影响显著，但多数技能效果欠佳 Testing skills significantly impact performance but many skills fail to deliver
• 代理常使用无效测试策略，如过度依赖随机输入或表面操作 Agents frequently use ineffective strategies like excessive random inputs
• 研究显示默认设置下代理表现优于部分测试技术 Default settings outperform many testing techniques in baseline performance

Takeaway: AI代理测试能力存在根本性缺陷，需更系统的方法提升其测试技术应用水平。 / AI agents exhibit fundamental limitations in testing capabilities requiring systemic improvements.