Paper Detail

Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents

Haoyu Zhao, Sitian Tu, Yuxuan Huang, Huichi Zhou, Zhangyue Yin, Jiahao Huang, Jingyuan Xu, Tianmiao Shi, Jin Shi, Nianxi Zhang, Yangmiao Zhang, Zhuolang Chan, Huiyan Hu, Chenchao Sheng, Yaqing Luo, Yue Chen, Huaxiu Yao, Meng Fang, Jun Wang

arxiv Score 13.3

Published 2026-10-25 · First seen 2026-10-27

Research Track A · General AI

Abstract

Continual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience. We introduce Continual Learning Exam (CLE), a large-scale benchmark developed by 50+ domain experts to measure experience-driven improvement in realistic professional workflows. CLE comprises 120 human-authored tasks, each organised as a five-level ladder, yielding 600 levels spanning 89 professional roles across 12 occupational domains. Agents must pass each level to unlock the next, with later levels building on experience accumulated during earlier assignments. An evaluation agent assesses each submission against human-authored rubrics. We compare a single-submission setting with one that provides diagnostic feedback after failure and allows one revision. Across 17 harness-model configurations using frontier LLMs, feedback and revision yield a median completion gain of 5.3 percentage points. Yet the best-performing system clears only 47.7% of levels and completes all five levels in just 23.3% of tasks, highlighting the difficulty of sustained learning from experience for existing frontier LLMs. As such, the dataset can also be used to post-train LLMs to strengthen their continual learning capabilities.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@article{zhao2026continual,
  title = {Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents},
  author = {Haoyu Zhao and Sitian Tu and Yuxuan Huang and Huichi Zhou and Zhangyue Yin and Jiahao Huang and Jingyuan Xu and Tianmiao Shi and Jin Shi and Nianxi Zhang and Yangmiao Zhang and Zhuolang Chan and Huiyan Hu and Chenchao Sheng and Yaqing Luo and Yue Chen and Huaxiu Yao and Meng Fang and Jun Wang},
  year = {2026},
  abstract = {Continual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience. We introduce Continual Learning Exam (CLE), a large-scale benchmark developed by 50+ domain experts to measure experience-driven improvement in },
  url = {https://arxiv.org/abs/2610.33956},
  keywords = {cs.LG, cs.AI, cs.CL, cs.CV},
  eprint = {2610.33956},
  archiveprefix = {arXiv},
}

Metadata

{}