Paper Detail

Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba, Adam Gleave, Chris Cundy

arxiv Score 7.8

Published 2026-10-08 · First seen 2026-10-09

General AI

Abstract

Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes achieve 98.8% AUC in SHADE-Arena, surpassing an Opus 5.5 text-monitoring baseline, and show improved efficacy as the underlying model is scaled up. To push our probes to their limit, we test them on several cases where deception cannot be determined from the context alone. In these cases, which we refer to as introspective deception, the ground truth can only be determined through careful elicitation or thorough knowledge of a model's training data. In one such evaluation, we show that probes can distinguish transcripts containing a model's true hidden goal from other goals with an AUC of up to 99.7%. Our probes also readily detect deception on prominent open-weight models which lie about politically sensitive topics, and about their beliefs when put under pressure. We release our training dataset, dubbed FIBS, to help drive frontier deployment of effective probes, and encourage the community to expand upon it with further examples of deception and sabotage.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@article{hollinsworth2026caught,
  title = {Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception},
  author = {Oskar J. Hollinsworth and Alex F. Spies and Tigist Diriba and Adam Gleave and Chris Cundy},
  year = {2026},
  abstract = {Recent incidents have highlighted the challenge of monitoring LLM agents and the danger of models deceiving people. We show that white-box deception detection via probes can be scaled up to frontier monitoring settings by collecting the largest deception dataset to date for training probes and introducing a novel probe architecture which can aggregate information across many layers and tokens. Our probes achieve 98.8\% AUC in SHADE-Arena, surpassing an Opus 5.5 text-monitoring baseline, and show },
  url = {https://arxiv.org/abs/2610.12445},
  keywords = {cs.LG, cs.AI, Deception, Context (archaeology), Computer science, Lie detection, Computer security},
  eprint = {2610.12445},
  archiveprefix = {arXiv},
}

Metadata

{}