Paper Detail
Xiangfan Wu, Zonghao Ying, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo
As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \textbf{Ventor-QTest}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports average fidelity loss (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report extreme fidelity loss (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@misc{wu2026ventor,
title = {Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs},
author = {Xiangfan Wu and Zonghao Ying and Huiyu Wu and Xing Zheng and Huangsheng Cheng and Xiaorong Shi and Jing Guo},
year = {2026},
abstract = {As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \textbackslash{}textbf\{Ventor-QTest\}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target m},
url = {https://huggingface.co/papers/2608.16391},
keywords = {Ventor-QTest, black-box audit, categorical output distribution, average fidelity loss, coarsened-KL, extreme fidelity loss, reference-centered-surprisal, GPQA-Diamond, Terminal-Bench, long-horizon agentic tasks, huggingface daily},
eprint = {2608.16391},
archiveprefix = {arXiv},
}
{}