Paper Detail
Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, Xiangyan Liu, Jiaqi Liao, Chonghe Jiang, Zhenglin Wan, Jiawei Gu, Pengfei Zhou, Rui Huang, Ziqi Zhao, Shengyuan Ding, Ailing Yu, Bo Peng, Bowei Xia, Hao Sun, Haotian Liang, Ji Xie, Jiajun Chen, Jiajun Song, Liu Yang, Ming Xu, Qionglin Qiu, Runhao Fu, Shengfang Zhai, Shijian Wang, Tengfei Ma, Tianyi Wu, Weiyang Jin, Yan Wang, Yang Dai, Yao Lai, Youwei Shu, Yue Liu, Yunzhuo Hao, Yuwei Niu, Jinkai Huang, Jiayuan Zhuo, Zhennan Shen, Linyu Wu, Cihang Xie, Yuyin Zhou, Jiaheng Zhang, Zeyu Zheng, Mengkang Hu, Michael Qizhe Shieh
Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain largely text-centric. We introduce , a benchmark for coworker agents built around multi-turn multi-day tasks, a stateful sandboxed service environment whose state evolves between turns, and rule-based verification. The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet) and scored by 1537 deterministic Python checkers over post-execution service state; no LLM-as-judge is invoked during scoring. We benchmark seven frontier agent systems. The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0\%, indicating that partial progress is common while complete end-to-end workflow completion remains rare. Turn-level analysis shows that performance drops after the first exogenous environment update, highlighting adaptation to changing state as a key open challenge. We release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@misc{meng2026clawmark,
title = {ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents},
author = {Fanqing Meng and Lingxiao Du and Zijian Wu and Guanzheng Chen and Xiangyan Liu and Jiaqi Liao and Chonghe Jiang and Zhenglin Wan and Jiawei Gu and Pengfei Zhou and Rui Huang and Ziqi Zhao and Shengyuan Ding and Ailing Yu and Bo Peng and Bowei Xia and Hao Sun and Haotian Liang and Ji Xie and Jiajun Chen and Jiajun Song and Liu Yang and Ming Xu and Qionglin Qiu and Runhao Fu and Shengfang Zhai and Shijian Wang and Tengfei Ma and Tianyi Wu and Weiyang Jin and Yan Wang and Yang Dai and Yao Lai and Youwei Shu and Yue Liu and Yunzhuo Hao and Yuwei Niu and Jinkai Huang and Jiayuan Zhuo and Zhennan Shen and Linyu Wu and Cihang Xie and Yuyin Zhou and Jiaheng Zhang and Zeyu Zheng and Mengkang Hu and Michael Qizhe Shieh},
year = {2026},
abstract = {Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain larg},
url = {https://huggingface.co/papers/2604.23781},
keywords = {language-model agents, multi-turn multi-day tasks, stateful sandboxed service environment, rule-based verification, agent systems, workflow completion, exogenous environment updates, code available, huggingface daily},
eprint = {2604.23781},
archiveprefix = {arXiv},
}
{}