Paper Detail

RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios

Chenhao Yuan, Bohan Yu, Yinhao Xu, Chao Yin, Chenxi Zhou, Zhaoxuan Wang, Shuwen Xu, Shaoping Huang, Haolin Ren, Jiaxiang Liu, Yao Xu, Jun Zhao, Kang Liu

huggingface Score 18.0

Published 2026-10-14 · First seen 2026-10-15

General AI

Abstract

Large language models (LLMs) increasingly tackle complex real-world tasks in specialized domains. However, identifying and correctly applying domain-specific rules remains challenging when scenario information is distributed across multiple files. Existing rule-reasoning benchmarks typically represent scenarios using only a few factual statements, simplifying the identification and application of relevant rules. To address this gap, this paper introduces RuleTrek, a multi-domain benchmark for agentic rule reasoning in complex scenarios. Its rule-grounded construction pipeline extracts structured rule graphs, generates and incrementally composes rule-specific sub-scenarios while ensuring consistency and solvability. To systematically evaluate agentic rule reasoning in complex scenarios, RuleTrek assesses three capabilities in interactive workspaces: Rule Identification (RI), Rule Deduction (RD), and End-to-End Rule Reasoning (E2E). Experiments on 11 frontier LLMs show that the highest RI and RD scores are only 62.47\% and 60.43\%, respectively, while E2E scores range from 26.97\% to 55.29\%. These results demonstrate that current LLM agents face substantial challenges in rule reasoning over complex scenarios. We make our code and dataset publicly available at https://github.com/yuan247/RuleTrek{RuleTrek}.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{yuan2026ruletrek,
  title = {RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios},
  author = {Chenhao Yuan and Bohan Yu and Yinhao Xu and Chao Yin and Chenxi Zhou and Zhaoxuan Wang and Shuwen Xu and Shaoping Huang and Haolin Ren and Jiaxiang Liu and Yao Xu and Jun Zhao and Kang Liu},
  year = {2026},
  abstract = {Large language models (LLMs) increasingly tackle complex real-world tasks in specialized domains. However, identifying and correctly applying domain-specific rules remains challenging when scenario information is distributed across multiple files. Existing rule-reasoning benchmarks typically represent scenarios using only a few factual statements, simplifying the identification and application of relevant rules. To address this gap, this paper introduces RuleTrek, a multi-domain benchmark for ag},
  url = {https://huggingface.co/papers/2610.19698},
  keywords = {RuleTrek, rule-grounded construction, structured rule graphs, rule-specific sub-scenarios, Rule Identification, Rule Deduction, End-to-End Rule Reasoning, LLM agents, code available, huggingface daily},
  eprint = {2610.19698},
  archiveprefix = {arXiv},
}

Metadata

{}