Paper Detail
Seongtae Hong, Youngjoon Jang, Jungseob Lee, Chanjun Park, Heuiseok Lim
Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive retrieval objective. By introducing trainable prefix positions that interact with regular tokens during encoding, [DUMP] tokens help reduce the influence of residual or uninformative token-level signals on the final retrieval representation. Experiments on standard retrieval and long-context benchmarks show that our method consistently improves over finetuned baselines across encoder and decoder embedding models. Further analysis shows that [DUMP] tokens alter token-level activation and attention patterns associated with low-information tokens. These results suggest that trainable prefix tokens provide a lightweight way to improve retrieval representations.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@misc{hong2026reducing,
title = {Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens},
author = {Seongtae Hong and Youngjoon Jang and Jungseob Lee and Chanjun Park and Heuiseok Lim},
year = {2026},
abstract = {Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive r},
url = {https://huggingface.co/papers/2611.07298},
keywords = {embedding models, fixed-size vector representations, attention sink behavior, trainable [DUMP] tokens, contrastive retrieval objective, trainable prefix positions, encoder embedding models, decoder embedding models, token-level activation, attention patterns, low-information tokens, retrieval representations, long-context benchmarks, huggingface daily},
eprint = {2611.07298},
archiveprefix = {arXiv},
}
{}