Paper Detail
Mona Schirmer, Metod Jazbec, Alexander Timans, Christian Naesseth, Maja Waldron, Eric Nalisnick
Despite alignment training, LLMs remain prone to generating unsafe outputs at deployment time. Monitoring outputs online and raising an alarm when safety can no longer be assumed is therefore critical. We study a simple real-time monitor that turns a verifier signal from an external model into an alarm decision by thresholding, with the threshold calibrated via risk control. In experiments on mathematical reasoning and red teaming datasets, we show that this simple design is competitive with more advanced monitors based on sequential hypothesis testing.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@article{schirmer2026online,
title = {Online Safety Monitoring for LLMs},
author = {Mona Schirmer and Metod Jazbec and Alexander Timans and Christian Naesseth and Maja Waldron and Eric Nalisnick},
year = {2026},
abstract = {Despite alignment training, LLMs remain prone to generating unsafe outputs at deployment time. Monitoring outputs online and raising an alarm when safety can no longer be assumed is therefore critical. We study a simple real-time monitor that turns a verifier signal from an external model into an alarm decision by thresholding, with the threshold calibrated via risk control. In experiments on mathematical reasoning and red teaming datasets, we show that this simple design is competitive with mor},
url = {https://arxiv.org/abs/2607.02510},
keywords = {cs.AI, cs.CL, cs.LG, stat.AP, stat.ML},
eprint = {2607.02510},
archiveprefix = {arXiv},
}
{}