Paper Detail

Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning

Xinjie Shen

huggingface Score 9.4

Published 2026-11-09 · First seen 2026-11-11

General AI

Abstract

Reinforcement learning (RL) can improve an agent's task capability while reinforcing behavior that violates contextual execution requirements. We study this reward-driven misalignment, including reward hacking, through verifiable reward--context conflicts, where compliant task completion remains feasible while some executable shortcuts earn more reward than every compliant strategy. Encoding these conflicts in task dynamics lets us generate diverse interactive environments at scale and derive compliance feedback from execution histories. In resource and reporting workflows, task optimization increases real productivity through unauthorized resource use or raises reported scores while actual work declines. Controlled changes in the training distribution show that minority conflict exposure can still lead to widespread shortcut acquisition, reaching 97.9\% certified profitable violations on an API task under a 25\% conflict mixture after 128 updates. The same contextual checks supply a simple binary penalty that supports joint capability and compliance gains and partially mitigates acquired violations. Learned safety adapts to expanded authorization and transfers across mechanisms into workflows with evolving evidence and approval requirements. On 32 unseen changing-evidence tasks, resource-trained safety increases safe completion from 53.1\% to 89.1\% relative to task-only training, while both policies achieve full real work. These results connect a verifiable account of reward-driven misalignment with scalable task construction and safety learning that remains useful under changing execution requirements.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{shen2026shortcuts,
  title = {Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning},
  author = {Xinjie Shen},
  year = {2026},
  abstract = {Reinforcement learning (RL) can improve an agent's task capability while reinforcing behavior that violates contextual execution requirements. We study this reward-driven misalignment, including reward hacking, through verifiable reward--context conflicts, where compliant task completion remains feasible while some executable shortcuts earn more reward than every compliant strategy. Encoding these conflicts in task dynamics lets us generate diverse interactive environments at scale and derive co},
  url = {https://huggingface.co/papers/2611.12865},
  keywords = {reward hacking, reward-driven misalignment, verifiable reward-context conflicts, compliant task completion, executable shortcuts, task dynamics, interactive environments, compliance feedback, execution histories, training distribution, conflict exposure, shortcut acquisition, certified profitable violations, API task, conflict mixture, contextual checks, binary penalty, joint capability and compliance gains, acquired violations, learned safety, expanded authorization, transfers across mechanisms, changing-evidence tasks, resource-trained safety, task-only training, huggingface daily},
  eprint = {2611.12865},
  archiveprefix = {arXiv},
}

Metadata

{}