Paper Detail
Shi Qiu, Junyi Deng, Yiwei Deng, Haoran Dong, Jieyu Fu, Mao Li, Zeyu Li, Zhaolong Zhang, Huiwen Zheng, Leidong Bao, Anqi Lv, Zihan Mo, Yadi Niu, Yiyang Peng, Yu Tian, Yili Wang, Ziyu Wang, Zi-Yu Wang, Jiashen Wei, Liuheng Wu, Aoran Xue, Leyi Yang, Guanglu Yuan, Xiarui Zhan, Jingjun Zhang, Zifan Zheng, Pengfei Liu, Linrui Zhen, Kaiyang Li, Qichang Li, Ziheng Zhou, Guo-En Nian, Yunwei Xiao, Qing-Hong Cao, Linjie Dai, Xu Feng, Peng Gao, Ying Gu, Chang Liu, Jia Liu, Ming-xing Luo, Yan-Qing Ma, Liang-You Peng, Huichao Song, Shufeng Wang, Chenxu Wang, Tao Wang, Yi-Nan Wang, Chengyin Wu, Pengwei Zhao, Hua Xing Zhu
AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published paper, implement the corresponding algorithms from scratch, and produce quantitative results matching the original publication. Agents are provided only with the task instruction and paper content, and operate in a sandboxed execution environment. All tasks are contributed by domain experts from over 20 research groups at the School of Physics, Peking University, each grounded in a real published paper and validated through end-to-end reproduction with verified ground-truth results and detailed scoring rubrics. Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution. The best-performing agent, OpenAI Codex powered by GPT-5.3-Codex, achieves a mean overall score of 34%. All agents exhibit a zero end-to-end callback success rate, with particularly poor performance in data accuracy and code correctness. We further identify systematic failure modes, including errors in formula implementation, inability to debug numerical simulations, and fabrication of output data. Overall, PRBench provides a rigorous benchmark for evaluating progress toward autonomous scientific research.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@misc{qiu2026prbench,
title = {PRBench: End-to-end Paper Reproduction in Physics Research},
author = {Shi Qiu and Junyi Deng and Yiwei Deng and Haoran Dong and Jieyu Fu and Mao Li and Zeyu Li and Zhaolong Zhang and Huiwen Zheng and Leidong Bao and Anqi Lv and Zihan Mo and Yadi Niu and Yiyang Peng and Yu Tian and Yili Wang and Ziyu Wang and Zi-Yu Wang and Jiashen Wei and Liuheng Wu and Aoran Xue and Leyi Yang and Guanglu Yuan and Xiarui Zhan and Jingjun Zhang and Zifan Zheng and Pengfei Liu and Linrui Zhen and Kaiyang Li and Qichang Li and Ziheng Zhou and Guo-En Nian and Yunwei Xiao and Qing-Hong Cao and Linjie Dai and Xu Feng and Peng Gao and Ying Gu and Chang Liu and Jia Liu and Ming-xing Luo and Yan-Qing Ma and Liang-You Peng and Huichao Song and Shufeng Wang and Chenxu Wang and Tao Wang and Yi-Nan Wang and Chengyin Wu and Pengwei Zhao and Hua Xing Zhu},
year = {2026},
abstract = {AI agents powered by large language models exhibit strong reasoning and problem-solving capabilities, enabling them to assist scientific research tasks such as formula derivation and code generation. However, whether these agents can reliably perform end-to-end reproduction from real scientific papers remains an open question. We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics. Each task requires an agent to comprehend the methodology of a published pap},
url = {https://huggingface.co/papers/2603.27646},
keywords = {large language models, scientific reasoning, end-to-end reproduction, formula derivation, code generation, benchmark, agentified assessment pipeline, coding agents, quantitative results, numerical simulations, huggingface daily},
eprint = {2603.27646},
archiveprefix = {arXiv},
}
{}