Paper Detail

Experience-Handoff Reinforcement Learning for Recursive Self-Improvement

Doyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang

huggingface Score 19.0

Published 2026-11-12 · First seen 2026-11-14

General AI

Abstract

It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering () harness and train one model to act, reflect, and curate task-local playbooks. Our method, , evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{kim2026experience,
  title = {Experience-Handoff Reinforcement Learning for Recursive Self-Improvement},
  author = {Doyoung Kim and Byeongguk Jeon and Kailin Zhang and Sherry Yang},
  year = {2026},
  abstract = {It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g.},
  url = {https://huggingface.co/papers/2611.16409},
  keywords = {Large Language Models, Reinforcement Learning, scalar rewards, agentic tasks, agent harness, reflection, compaction, rollouts, gradient updates, Agentic Context Engineering, experience-handoff artifacts, task-local playbooks, action generator, insight reflector, playbook curator, fixed scaffold, agentic benchmarks, code available, huggingface daily},
  eprint = {2611.16409},
  archiveprefix = {arXiv},
}

Metadata

{}