· 6 min read

all-MiniLM-L6-v2 Max Sequence Length: 256 Tokens, Not 512

all-MiniLM-L6-v2 truncates at 256 tokens, about 900 characters, and drops the rest without a warning. Why the configs say 128, 256 and 512, what chunk size to use, and why raising max_seq_length to 512 makes retrieval worse.

Three files in the model repository disagree about how long an input can be, and the one that wins is not the one most people assume.

from sentence_transformers import SentenceTransformer
from huggingface_hub import hf_hub_download
import json

m = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
tj = json.load(open(hf_hub_download("sentence-transformers/all-MiniLM-L6-v2",
                                    "tokenizer.json")))

print(tj["truncation"]["max_length"])                       # 128
print(m.max_seq_length)                                     # 256
print(m[0].auto_model.config.max_position_embeddings)       # 512

encode passes max_length=self.max_seq_length on every call, so the tokenizer's 128 gets overridden and the model's 512 is never reached. 256 wins. For all-mpnet-base-v2 it is 384.

It truncates without telling you

Same text, same model, only the window changes:

body = ("The retrieval service keeps an inverted index in memory and refreshes it "
        "nightly from the document store. Queries are served from that snapshot, so "
        "writes are not visible until the next refresh completes. ")
doc = body * 12 + "The rollback procedure is documented under runbook QX-4417."
print(len(m.tokenizer(doc, truncation=False)["input_ids"]))   # 509

m.max_seq_length = 256
a256 = m.encode(doc, normalize_embeddings=True)
m.max_seq_length = 512
a512 = m.encode(doc, normalize_embeddings=True)

print(a256 @ a512)                                            # 0.9359

Two different vectors for the same document. No warning, no error, no truncation flag in the output. At the default 1000-character chunk size this is the normal case, not an edge case:

lengths = [len(m.tokenizer(c, truncation=False)["input_ids"]) for c in chunks]
print(sum(l > 256 for l in lengths), "/", len(lengths))       # 85 / 98

LangChain's RAG tutorial splits with RecursiveCharacterTextSplitter at chunk_size=1000, so on that setup nine chunks in ten are longer than the model reads. The class default, 4000, is worse. English runs about 3.5 characters per token, so 256 tokens is roughly 900 characters and a 1000-character chunk sits just past the edge.

A longer window is not the fix

The document above has its distinctive fact in the tail, past the 256 mark. Two queries, one for the tail and one for the head:

tail = "what is the rollback runbook number"
head = "how often is the inverted index refreshed"

# tail: 256 -> 0.1239   512 -> 0.2049
# head: 256 -> 0.5154   512 -> 0.4457

The tail improving is expected. The head getting worse is the part that matters: these models mean-pool, so 250 more tokens of unrelated content drag the centroid away from whatever the query was about. A longer window buys recall on the tail and pays for it in precision everywhere else.

Chunk instead, and score by the best chunk

import numpy as np

qv = m.encode(tail, normalize_embeddings=True)
cv = np.stack([m.encode(c, normalize_embeddings=True) for c in chunks])  # 3 chunks
best = (cv @ qv).max()

# tail: truncated 0.1239   best-chunk 0.4789
# head: truncated 0.5154   best-chunk 0.5058

The tail goes from noise to 0.48, more than double what the 512 window gave, and the head holds. That is the whole argument against reaching for nomic at 8192 or bge-m3: a chunk that fits under the limit gets the tail and keeps its focus, while a chunk that needs 8192 tokens averages too many things to match any of them sharply.

So chunk by tokens rather than characters, since code and tables and non-English text all have different ratios. Target well under the limit, 200 tokens with overlap never truncates. Then take the maximum similarity across a document's chunks instead of embedding it whole.

Checking your own corpus

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")

lengths = [len(tok(c, truncation=False)["input_ids"]) for c in your_chunks]
over = sum(l > 256 for l in lengths)
print(f"{over}/{len(lengths)} chunks exceed 256 tokens")

If that is not close to zero, part of your corpus is not in your index and nothing in your pipeline will tell you.

512 is a hard ceiling

There are exactly 512 learned position embeddings in the weights. Ask for more and a naive implementation has nothing to add for the extra positions, so you get either an index error or tokens entering attention with no positional information, which returns a normal looking vector that is quietly wrong.

What Kjarni does about it

I hit this building Kjarni, a native inference engine. Its encoders read max_position_embeddings and truncated at 512. Parity against PyTorch passed, because every parity test used short sentences: the longest string in the entire encoder suite was 83 characters. So it matched the reference exactly right up until an input crossed 256 tokens, which the default chunk size put 87% of chunks above.

Kjarni now reads sentence_bert_config.json when the model ships one, the same file sentence-transformers reads, so the truncation point matches whatever your Python pipeline already does. Asking for more positions than the weights hold is an error rather than a silently wrong vector.

kjarni embed "your text" --model minilm-l6-v2

The wider lesson was about the tests, not the config. A parity suite whose longest input is 83 characters proves the model loads, and nothing about the case that actually broke.

FAQ

What is the max sequence length of all-MiniLM-L6-v2?

256 tokens. That is max_seq_length in sentence_bert_config.json, and it is what sentence-transformers passes to the tokenizer on every encode call. The 512 in config.json is how many position embeddings the weights have, and the 128 in tokenizer.json is overridden. Anything past 256 tokens is dropped.

What chunk size should I use with all-MiniLM-L6-v2?

Around 200 tokens, measured with the model's own tokenizer, with some overlap. In characters that is roughly 700 for English prose, but code, tables and other languages tokenize differently, so counting characters is only an estimate. 256 tokens is about 900 characters, which is why the common 1000-character chunk is just over the limit.

Does sentence-transformers warn when it truncates?

No. encode truncates to max_seq_length and returns a normal vector. There is no warning, error or flag in the output. Count tokens with tokenizer(text, truncation=False) to find out which inputs were cut.

Can I set max_seq_length to 512?

You can, and it runs, because the weights have 512 positions. It does not help: the model was trained at shorter lengths and mean-pools every token, so a longer input averages more topics together. In the measurement above, 512 improved a query about the tail of a document and made a query about its head worse. Chunking improved both. The same holds when the setting is changed through fastembed, LangChain or any other wrapper, since the weights are the same.

Does KeyBERT truncate at 256 tokens too?

Yes, by default. KeyBERT's default model is all-MiniLM-L6-v2, loaded through sentence-transformers, so a document longer than 256 tokens is embedded from its first 256 only. Keywords that appear only later in the document are compared against an embedding that never saw them. Split long documents before extracting keywords.

What are the limits of other sentence-transformers models?

all-mpnet-base-v2 reads 384 tokens. all-MiniLM-L12-v2 and paraphrase-multilingual-MiniLM-L12-v2, the usual pick for German and other non-English text, read only 128. Check max_seq_length in each model's sentence_bert_config.json rather than assuming a model reads as far as its position embeddings go.


Kjarni is a native inference engine for embeddings, search, reranking and chat, with bindings for C#, the browser, Go and Python. The source is on GitHub, and there is a browser demo that needs no install.

Kjarni runs the same engine on every platform: