Paper Detail

Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens

Seongtae Hong, Youngjoon Jang, Jungseob Lee, Chanjun Park, Heuiseok Lim

huggingface Score 10.4

Published 2026-11-05 · First seen 2026-11-07

General AI

Abstract

Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive retrieval objective. By introducing trainable prefix positions that interact with regular tokens during encoding, [DUMP] tokens help reduce the influence of residual or uninformative token-level signals on the final retrieval representation. Experiments on standard retrieval and long-context benchmarks show that our method consistently improves over finetuned baselines across encoder and decoder embedding models. Further analysis shows that [DUMP] tokens alter token-level activation and attention patterns associated with low-information tokens. These results suggest that trainable prefix tokens provide a lightweight way to improve retrieval representations.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{hong2026reducing,
  title = {Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens},
  author = {Seongtae Hong and Youngjoon Jang and Jungseob Lee and Chanjun Park and Heuiseok Lim},
  year = {2026},
  abstract = {Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive r},
  url = {https://huggingface.co/papers/2611.07298},
  keywords = {embedding models, fixed-size vector representations, attention sink behavior, trainable [DUMP] tokens, contrastive retrieval objective, trainable prefix positions, encoder embedding models, decoder embedding models, token-level activation, attention patterns, low-information tokens, retrieval representations, long-context benchmarks, huggingface daily},
  eprint = {2611.07298},
  archiveprefix = {arXiv},
}

Metadata

{}