A Dataset for Modeling Iterative Problem-Solving

AI Digest

本文提出CodeInsight数据集，包含300万次编程提交，用于研究迭代问题解决过程。通过对比模型预测效果，发现LLM更适合作为生成器而非行为预测器。

This paper introduces CodeInsight, a dataset of 3M+ programming submissions, to study iterative problem-solving. Model comparisons reveal LLMs better serve as generators than behavior predictors.

Key points

• 构建CodeInsight数据集，收录2门C++课程3286名学生的300万次提交 Created CodeInsight dataset with 3.28M submissions from 2 C++ courses
• 开发基准测试评估参数化、序列化和生成式模型的预测能力 Developed benchmark to evaluate parametric/sequential/generative models
• 发现LLM生成器表现优于行为预测模型，但预测精度较低 LLMs show better generation ability than behavior prediction accuracy
• 通过离散潜变量追踪求解者特征的RSSM模型表现最佳 RSSM with discrete latent variables achieves best prediction

Takeaway: 迭代问题解决研究需大规模真实数据，模型选择应区分生成与预测功能。 / Iterative problem-solving research needs large-scale real data; model selection should distinguish generation from prediction functions.

Why it matters 提供真实迭代数据与多模型对比，对教育AI和自主系统研究有实践价值。