Paper Detail

ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty

Yuyang Dai, Yuting Tang, Futoshi Futami, Takashi Ishida

huggingface Score 6.0

Published 2026-10-12 · First seen 2026-10-15

General AI

Abstract

Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{dai2026probabilitybench,
  title = {ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty},
  author = {Yuyang Dai and Yuting Tang and Futoshi Futami and Takashi Ishida},
  year = {2026},
  abstract = {Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute bo},
  url = {https://huggingface.co/papers/2610.17016},
  keywords = {model uncertainty, conditional class probabilities, ProbabilityBench, Bayes error, KL divergence, calibration metrics, decision models, Jev, Perplexity Decider, code available, huggingface daily},
  eprint = {2610.17016},
  archiveprefix = {arXiv},
}

Metadata

{}