DolphinBench: Mapping the Pareto Frontier of Agent Memory

AI Digest

DolphinBench提出新基准，通过任务完成评估代理记忆，强调成本与延迟，弥补现有基准仅关注准确率的不足。

DolphinBench introduces a benchmark evaluating agent memory through task completion, emphasizing cost and latency alongside accuracy, addressing gaps in existing benchmarks.

Key points

• 现有基准多用于问答格式，忽视实际任务中的记忆需求 Existing benchmarks focus on QA formats, neglecting memory needs in real tasks
• DolphinBench通过3个知识角色验证200个任务，强制依赖历史信息 DolphinBench validates 200 tasks across 3 personas requiring historical context
• 首次要求评估报告总成本和延迟，实现综合性能分析 First benchmark requiring cost/latency reports alongside accuracy
• 数据集与代码公开，便于复现与改进 Public dataset and code enable reproducibility and improvement

Takeaway: 该基准为优化记忆系统提供新评估维度。 / This benchmark offers new evaluation dimensions for optimizing memory systems.

Why it matters 填补记忆系统评估空白，为实际应用提供量化指导。