Paper Detail

Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference

Saul Urso, Emanuele Carlini, Patrizio Dazzi, Matteo Mordacchini

huggingface Score 11.4

Published 2026-11-06 · First seen 2026-11-12

General AI

Abstract

Large language models (LLMs) increasingly support Web-based services that must balance response quality and inference cost across heterogeneous requests. Conventional routing selects one model per prompt, while fixed ensembles cannot adapt their composition and cost to individual requests. We introduce PISA, a prompt-adaptive, cost-aware single-model router, and PISA-E, its extension to ensemble routing. At the core of both methods, PISA combines soft cluster memberships with model-specific predictors to estimate individual model capabilities. Building on these estimates, PISA-E jointly selects ensemble composition and cardinality under an explicit cost--quality preference that can be adjusted at inference time without retraining. A tractable ensemble-success surrogate and efficient candidate selection enable this extension without ensemble-specific training. Across four LLM routing benchmarks under in-distribution (ID) and out-of-distribution (OOD) conditions, PISA achieves the highest macro-average cost--performance hypervolume among the evaluated single-model routers in both settings. Meanwhile, PISA-E achieves the highest macro-average task performance among the evaluated ensemble approaches when prioritizing response quality, surpassing the strongest competitors by 2.09 and 2.87 percentage points under ID and OOD conditions, respectively. At selected lower-cost operating points, it also exceeds the strongest ensemble competitors' macro-average task performance while reducing average inference cost by approximately 36% and 25%, respectively. These results demonstrate the effectiveness of prompt-adaptive capability estimation for both single-model routing and selective ensembling.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{urso2026ensemble,
  title = {Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference},
  author = {Saul Urso and Emanuele Carlini and Patrizio Dazzi and Matteo Mordacchini},
  year = {2026},
  abstract = {Large language models (LLMs) increasingly support Web-based services that must balance response quality and inference cost across heterogeneous requests. Conventional routing selects one model per prompt, while fixed ensembles cannot adapt their composition and cost to individual requests. We introduce PISA, a prompt-adaptive, cost-aware single-model router, and PISA-E, its extension to ensemble routing. At the core of both methods, PISA combines soft cluster memberships with model-specific pred},
  url = {https://huggingface.co/papers/2611.10193},
  keywords = {large language models (LLMs), prompt-adaptive routing, cost-aware single-model router, ensemble routing, soft cluster memberships, model-specific predictors, ensemble composition, ensemble cardinality, cost-quality preference, ensemble-success surrogate, candidate selection, in-distribution (ID), out-of-distribution (OOD), macro-average cost-performance hypervolume, single-model routers, ensemble approaches, inference cost, huggingface daily},
  eprint = {2611.10193},
  archiveprefix = {arXiv},
}

Metadata

{}