Paper Detail
Víctor Gallego
Coding agents can synthesize and optimize code-based policies from simulator feedback, allowing the resulting program to act with zero inference model calls. But for long horizon tasks, two challenges arise: local improvements can hurt final success and switching to a new objective requires another expensive synthesis run. We propose a framework that first develops competence by refining the policy through a feedback cascade, moving from episode returns to hierarchical goals and belief updates which can be scored densely from simulator. In the Craftax game, a partially observed and procedurally generated environment, refinements via episode level feedback stall at <1% final-level arrival, whereas with the feedback cascade the optimized policy reaches the final stage 50% of time (28% final boss defeats). Furthermore, to make this competence reusable towards new goals, we factor the previous controller into an environment interpreter I (models, memory, skills) and a rule program p_0, and we show that coding agents can optimize a reusable goal compiler C_θ that maps structured tasks g into new rule programs p_g. Across five independent compiler searches trained only on short tasks, the resulting compilers successfully transfer zero-shot to longer tasks without any task-specific tuning nor inference-time LLM calls: continuation-scored feedback and a fixed machine with an editable program together turn a single-task competent policy into a reusable one. Code at: https://github.com/vicgalle/craftax-program-policies
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@misc{gallego2026continuation,
title = {Continuation-Guided LLM Evolution of Program Policies and Goal Compilers in Craftax},
author = {Víctor Gallego},
year = {2026},
abstract = {Coding agents can synthesize and optimize code-based policies from simulator feedback, allowing the resulting program to act with zero inference model calls. But for long horizon tasks, two challenges arise: local improvements can hurt final success and switching to a new objective requires another expensive synthesis run. We propose a framework that first develops competence by refining the policy through a feedback cascade, moving from episode returns to hierarchical goals and belief updates w},
url = {https://huggingface.co/papers/2611.19537},
keywords = {coding agents, code-based policies, simulator feedback, zero inference model calls, long-horizon tasks, feedback cascade, episode returns, hierarchical goals, belief updates, Craftax, partially observed environment, procedurally generated environment, environment interpreter, rule program, goal compiler, structured tasks, zero-shot transfer, inference-time LLM calls, continuation-scored feedback, editable program, code available, huggingface daily},
eprint = {2611.19537},
archiveprefix = {arXiv},
}
{}