QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge
AI Digest
QuranicMMLU基准语言学维度生成式AI评估
QuranicMMLU提出新基准测试，评估生成式AI在古兰经语言知识上的表现，覆盖音系、形态学等五维，发现专用模型表现最佳但选择题评分可能掩盖问题。
Introduces QuranicMMLU, a benchmark evaluating generative AI on Quranic Arabic across five linguistic dimensions. Finds specialized models perform best, but multiple-choice scoring may mask flaws.
Key points
- 构建涵盖音系/形态学/句法等五维的QuranicMMLU基准 Develops QuranicMMLU benchmark covering five linguistic dimensions
- 生成区分认知层级与经文复杂度的980道人工审核问题 Generates 980 human-reviewed questions stratified by cognitive levels
- 发现选择题准确率(84%)高于开放题质量(60%)且排名高度相关 Multiple-choice accuracy (84%) outperforms open-ended quality (60%)
Takeaway: QuranicMMLU为评估AI在古兰经语言知识提供严谨框架 / QuranicMMLU offers a rigorous framework for evaluating Arabic NLP in Quranic domain
Why it matters 为阿拉伯NLP研究提供新基准，揭示评估方法差异对结果的影响
View original ↗ Back to hot list
This page is an aggregated digest from arxiv; content and hot-score data come from public sources. Copyright belongs to the original authors. We link to originals with nofollow and never republish full text.