Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models

AI Digest

本文提出Pinocchio方法，为闭源语言模型提供无需访问内部数据的快速不确定性估计，通过跨模型训练实现高准确率。

Pinocchio offers fast uncertainty estimation for black-box LLMs without model access, achieving 0.862 AUROC via cross-model training.

Key points

• 解决闭源API模型无法获取log概率和微调权限的不确定性估计难题 Addresses uncertainty estimation for closed-source LLMs lacking log-prob access
• 通过联合七种LLM响应训练，实现跨模型零样本迁移能力 Achieves cross-model zero-shot transfer via joint training on seven LLMs
• 仅需单次前向计算即可生成不确定性估计，无需访问模型内部状态 Generates uncertainty estimates with single forward pass only
• 轻量级0.8B模型表现接近大型模型，代码集成仅需两行 Lightweight 0.8B model matches large model performance with minimal code

Takeaway: Pinocchio为工业界提供了实用的黑盒模型可靠性评估方案。 / Pinocchio provides practical reliability assessment for industrial black-box models.

Why it matters 该方法解决了工业界无法访问模型内部数据的核心痛点，具有实际部署价值。