OpenAI 开源思维链可监控性评测数据集与代码
OpenAI 开源其思维链可监控性评测的数据集与参考代码,仓库地址为 https://github.com/openai/monitorability-evals,并同步发布计算 g-mean2 指标的代码和模拟评测脚手架。OpenAI 表示已在 GPT-5.4 thinking、GPT-5.2 thinking、GPT-5 thinking 和 o3 的系统卡中回填完整可监控性评测结果,未来前沿推理模型系统卡也会报告该结果。部分依赖私有或受限数据的评测未纳入开源,包括 Health Queries、FrontierMath、Math Diamond、OAI-Repo-Sabotage、Anti-Scheming 等。
puting our monitorability metric, g-mean2, along with a scaffold that simulates running evaluations. The scaffold is a high-level orchestration layer that loads dataset and prompt metadata, samples and grades rows using mocked model and monitor outputs, computes intervention, process, and outcome metrics, and writes samples and summaries to disk. We will not be supporting this scaffold because it is only meant to be an illustration of how to run our evaluations rather than a full evaluation fram
匹配字段正文
