all-MiniLM-L6-v2 Max Sequence Length: 256 Tokens, Not 512
all-MiniLM-L6-v2 truncates at 256 tokens, about 900 characters, and drops the rest without a warning. Why the configs say 128, 256 and 512, what chunk size to use, and why raising max_seq_length to 512 makes retrieval worse.
Three files in the model repository disagree about how long an input can be, and the one that wins is not the one most people assume.
from sentence_transformers import SentenceTransformer
from huggingface_hub import hf_hub_download
import json
m = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
tj = json.load(open(hf_hub_download("sentence-transformers/all-MiniLM-L6-v2",
"tokenizer.json")))
print(tj["truncation"]["max_length"]) # 128
print(m.max_seq_length) # 256
print(m[0].auto_model.config.max_position_embeddings) # 512
encode passes max_length=self.max_seq_length on every call, so the tokenizer's
128 gets overridden and the model's 512 is never reached. 256 wins. For
all-mpnet-base-v2 it is 384.
It truncates without telling you
Same text, same model, only the window changes:
body = ("The retrieval service keeps an inverted index in memory and refreshes it "
"nightly from the document store. Queries are served from that snapshot, so "
"writes are not visible until the next refresh completes. ")
doc = body * 12 + "The rollback procedure is documented under runbook QX-4417."
print(len(m.tokenizer(doc, truncation=False)["input_ids"])) # 509
m.max_seq_length = 256
a256 = m.encode(doc, normalize_embeddings=True)
m.max_seq_length = 512
a512 = m.encode(doc, normalize_embeddings=True)
print(a256 @ a512) # 0.9359
Two different vectors for the same document. No warning, no error, no truncation flag in the output. At the default 1000-character chunk size this is the normal case, not an edge case:
lengths = [len(m.tokenizer(c, truncation=False)["input_ids"]) for c in chunks]
print(sum(l > 256 for l in lengths), "/", len(lengths)) # 85 / 98
LangChain's RAG tutorial splits with RecursiveCharacterTextSplitter at
chunk_size=1000, so on that setup nine chunks in ten are longer than the model
reads. The class default, 4000, is worse. English runs
about 3.5 characters per token, so 256 tokens is roughly 900 characters and a
1000-character chunk sits just past the edge.
A longer window is not the fix
The document above has its distinctive fact in the tail, past the 256 mark. Two queries, one for the tail and one for the head:
tail = "what is the rollback runbook number"
head = "how often is the inverted index refreshed"
# tail: 256 -> 0.1239 512 -> 0.2049
# head: 256 -> 0.5154 512 -> 0.4457
The tail improving is expected. The head getting worse is the part that matters: these models mean-pool, so 250 more tokens of unrelated content drag the centroid away from whatever the query was about. A longer window buys recall on the tail and pays for it in precision everywhere else.
Chunk instead, and score by the best chunk
import numpy as np
qv = m.encode(tail, normalize_embeddings=True)
cv = np.stack([m.encode(c, normalize_embeddings=True) for c in chunks]) # 3 chunks
best = (cv @ qv).max()
# tail: truncated 0.1239 best-chunk 0.4789
# head: truncated 0.5154 best-chunk 0.5058
The tail goes from noise to 0.48, more than double what the 512 window gave, and the head holds. That is the whole argument against reaching for nomic at 8192 or bge-m3: a chunk that fits under the limit gets the tail and keeps its focus, while a chunk that needs 8192 tokens averages too many things to match any of them sharply.
So chunk by tokens rather than characters, since code and tables and non-English text all have different ratios. Target well under the limit, 200 tokens with overlap never truncates. Then take the maximum similarity across a document's chunks instead of embedding it whole.
Checking your own corpus
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
lengths = [len(tok(c, truncation=False)["input_ids"]) for c in your_chunks]
over = sum(l > 256 for l in lengths)
print(f"{over}/{len(lengths)} chunks exceed 256 tokens")
If that is not close to zero, part of your corpus is not in your index and nothing in your pipeline will tell you.
512 is a hard ceiling
There are exactly 512 learned position embeddings in the weights. Ask for more and a naive implementation has nothing to add for the extra positions, so you get either an index error or tokens entering attention with no positional information, which returns a normal looking vector that is quietly wrong.
What Kjarni does about it
I hit this building Kjarni, a native inference engine. Its
encoders read max_position_embeddings and truncated at 512. Parity against
PyTorch passed, because every parity test used short sentences: the longest string
in the entire encoder suite was 83 characters. So it matched the reference exactly
right up until an input crossed 256 tokens, which the default chunk size put 87% of
chunks above.
Kjarni now reads sentence_bert_config.json when the model ships one, the same
file sentence-transformers reads, so the truncation point matches whatever your
Python pipeline already does. Asking for more positions than the weights hold is an
error rather than a silently wrong vector.
kjarni embed "your text" --model minilm-l6-v2
The wider lesson was about the tests, not the config. A parity suite whose longest input is 83 characters proves the model loads, and nothing about the case that actually broke.
FAQ
What is the max sequence length of all-MiniLM-L6-v2?
256 tokens. That is max_seq_length in sentence_bert_config.json, and it is what
sentence-transformers passes to the tokenizer on every encode call. The 512 in
config.json is how many position embeddings the weights have, and the 128 in
tokenizer.json is overridden. Anything past 256 tokens is dropped.
What chunk size should I use with all-MiniLM-L6-v2?
Around 200 tokens, measured with the model's own tokenizer, with some overlap. In characters that is roughly 700 for English prose, but code, tables and other languages tokenize differently, so counting characters is only an estimate. 256 tokens is about 900 characters, which is why the common 1000-character chunk is just over the limit.
Does sentence-transformers warn when it truncates?
No. encode truncates to max_seq_length and returns a normal vector. There is no
warning, error or flag in the output. Count tokens with
tokenizer(text, truncation=False) to find out which inputs were cut.
Can I set max_seq_length to 512?
You can, and it runs, because the weights have 512 positions. It does not help: the model was trained at shorter lengths and mean-pools every token, so a longer input averages more topics together. In the measurement above, 512 improved a query about the tail of a document and made a query about its head worse. Chunking improved both. The same holds when the setting is changed through fastembed, LangChain or any other wrapper, since the weights are the same.
Does KeyBERT truncate at 256 tokens too?
Yes, by default. KeyBERT's default model is all-MiniLM-L6-v2, loaded through
sentence-transformers, so a document longer than 256 tokens is embedded from its
first 256 only. Keywords that appear only later in the document are compared against
an embedding that never saw them. Split long documents before extracting keywords.
What are the limits of other sentence-transformers models?
all-mpnet-base-v2 reads 384 tokens. all-MiniLM-L12-v2 and
paraphrase-multilingual-MiniLM-L12-v2, the usual pick for German and other
non-English text, read only 128. Check max_seq_length in each model's
sentence_bert_config.json rather than assuming a model reads as far as its
position embeddings go.
Kjarni is a native inference engine for embeddings, search, reranking and chat, with bindings for C#, the browser, Go and Python. The source is on GitHub, and there is a browser demo that needs no install.
Related
- Local Embeddings for Microsoft.Extensions.AI: using these models from .NET
- Build a Document Search Engine in C#: where chunk size shows up
- Reranking in C#: a cross-encoder reads query and document together, so it avoids the pooling problem above
Kjarni runs the same engine on every platform: