Paper Detail

Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization

Christian Moya, Purav Matlia, Elliott Thornley, Guang Lin

arxiv Score 5.3

Published 2026-10-26 · First seen 2026-10-27

General AI

Abstract

Preference optimization can learn spurious correlations, and augmenting the preference data with ties (equally preferred responses) can reduce spurious reliance. However, ties differ in their corrective effect, making their selection for augmentation important. For log-linear policies, we derive each tie's influence on spurious reliance and show that taking the largest predicted reductions is first-order optimal among subsets of equal size. Computing these influences requires knowledge of the spurious features and the curvature of the training objective, which is often unavailable or expensive to estimate. To this end, we explore using the magnitude of a tie's implicit reward margin and characterize the conditions under which it is a useful proxy for predicted influence. To test the theory and its qualitative transfer to neural models, we estimate influence with a counterfactual proxy for spurious reliance and use a generalized Gauss--Newton approximation to curvature for language models. Across log-linear policies, neural models, and large language models, our selection rules consistently achieve comparable reductions in spurious reliance with fewer ties than uniform selection.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@article{moya2026selective,
  title = {Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization},
  author = {Christian Moya and Purav Matlia and Elliott Thornley and Guang Lin},
  year = {2026},
  abstract = {Preference optimization can learn spurious correlations, and augmenting the preference data with ties (equally preferred responses) can reduce spurious reliance. However, ties differ in their corrective effect, making their selection for augmentation important. For log-linear policies, we derive each tie's influence on spurious reliance and show that taking the largest predicted reductions is first-order optimal among subsets of equal size. Computing these influences requires knowledge of the sp},
  url = {https://arxiv.org/abs/2610.35787},
  keywords = {cs.LG},
  eprint = {2610.35787},
  archiveprefix = {arXiv},
}

Metadata

{}