· 5 min read

Local Semantic Search in Python

Index a directory, then search it by keyword, by meaning, or both, from Python. One pip install with no dependencies, running on your own machine, with a worked example of where hybrid search goes wrong and what fixes it.

Most Python search examples stop at cosine similarity over a list. That works until the list stops fitting in memory, or until someone searches for an exact product code and discovers that embeddings do not do exact.

Kjarni ships an index instead: BM25 and vectors in the same store, on disk, queryable three ways. The engine is Rust, the package is pip install kjarni, and it has no dependencies.

from kjarni import Embedder

e = Embedder("minilm-l6-v2")
print(e.dim)                                     # 384
print(e.similarity("capital of Iceland",
                   "Reykjavik is the capital of Iceland"))
384
0.8984

numpy is imported on demand by encode_batch and falls back to lists when it is absent, so the base install pulls in nothing at all.

Indexing a directory

The examples below search four one-line files in a docs/ folder. Create them to follow along:

mkdir -p docs
echo "Iceland is a Nordic island country in the North Atlantic, with a population of about 390,000. Around two thirds of the population is in the capital region around Reykjavik." > docs/iceland.txt
echo "Norway is a Nordic country on the Scandinavian peninsula. Oslo is the capital and the most populous city in the country." > docs/norway.txt
echo "The Icelandic krona is the currency of Iceland. Coins and banknotes are issued by the Central Bank of Iceland." > docs/currency.txt
echo "Python is a high-level programming language known for its readable syntax and large standard library." > docs/python.txt
from kjarni import Indexer

indexer = Indexer(model="minilm-l6-v2")
stats = indexer.create("my_index", ["docs/"])
print(stats)
IndexStats(documents_indexed=4, chunks_created=4, dimension=384,
           size_bytes=11328, files_processed=4, files_skipped=0, elapsed_ms=22)

The index is a directory on disk. Nothing is hosted, nothing listens on a port, and the next process to open it just reads it.

Files are discovered, split into chunks and embedded in one pass. chunk_size, chunk_overlap, extensions and exclude_patterns are constructor arguments, and long runs take an on_progress callback and a CancelToken so a user can stop an import.

from kjarni import Searcher, SearchMode

s = Searcher(model="minilm-l6-v2")
hits = s.search("my_index", "Where do most Icelanders live?",
                mode=SearchMode.SEMANTIC, top_k=3)
for h in hits:
    print(f"{h.score:8.4f}  {h.text[:50]}")

The interesting part is what the three modes actually return for that query, over four short documents about Iceland, Norway, the Icelandic krona, and Python:

KEYWORD      1.2155  norway.txt
SEMANTIC     0.6848  iceland.txt     0.4412  currency.txt   0.3982  norway.txt
HYBRID       0.0323  norway.txt      0.0164  iceland.txt    0.0161  currency.txt

Three different scales, because they are three different things. BM25 scores term overlap. Semantic scores cosine distance between embeddings. Hybrid fuses the two rankings with reciprocal rank fusion, which produces small numbers by construction and is comparable only against itself.

Where hybrid goes wrong

Look at that table again. Semantic gets it right and hybrid gets it wrong.

BM25 returned exactly one document, and it was the wrong one. Remove the word "most" from the query and it returns nothing at all:

'Where do most Icelanders live?'   →  norway.txt 1.216
'Where do Icelanders live?'        →  (no matches)
'Iceland population'               →  iceland.txt 2.086, currency.txt 0.985

The only term BM25 matched was "most", which appears in "the most populous city in the country". "Icelanders" never matches "Iceland", because the tokenizer does not stem one to the other. BM25 is working correctly, and it is working correctly on a word that carries no meaning for this query.

Reciprocal rank fusion does not know that. It sees a document ranked first by one retriever and promotes it. One noisy match in the keyword channel is enough to outrank the document that actually answers the question.

This is worth knowing before you reach for hybrid search by default. It is usually better than either half, and it inherits the failure modes of both.

What fixes it

A cross-encoder reads the query and the document together rather than comparing two vectors computed separately, so it can tell that a document about Norway does not answer a question about Iceland:

s = Searcher(model="minilm-l6-v2",
             rerank_model="minilm-l6-v2-cross-encoder")

hits = s.search("my_index", "Where do most Icelanders live?",
                mode=SearchMode.HYBRID, top_k=3, rerank=True)
 4.8958  iceland.txt
-7.0531  currency.txt
-8.3266  norway.txt

Right answer first, and the gap is not subtle. Those are raw logits, the same numbers a cross-encoder produces under PyTorch: negative is normal, and only the ordering carries meaning. Pass them through a sigmoid if you need something between zero and one.

The cost is a second model pass over the candidates, which is why it runs on the top few rather than the whole index. Retrieve broadly, rerank narrowly.

Note that once a rerank_model is configured, reranking is on by default for every search. That is convenient and it also means the scores you see are the cross-encoder's, not the retriever's, in every mode. Pass rerank=False when you want to see what retrieval alone did.

Reranking on its own

The reranker is usable without an index, over any list of strings you already have:

from kjarni import Reranker

documents = [
    "Python is a high-level programming language.",
    "Reykjavik is the capital and largest city of Iceland.",
    "Oslo is the capital of Norway.",
]

r = Reranker("minilm-l6-v2-cross-encoder")
for hit in r.rerank_top_k("What is the capital of Iceland?", documents, 3):
    print(f"{hit.score:8.3f}  {hit.document}")
   8.342  Reykjavik is the capital and largest city of Iceland.
  -5.096  Oslo is the capital of Norway.
 -11.136  Python is a high-level programming language.

RerankResult carries index, score and document, so you can map back to whatever the strings came from.

When something is wrong

Model names are checked against the registry, and a wrong one tells you so:

>>> Reranker("ms-marco-minilm-l6-v2")
KjarniException: Unknown model 'ms-marco-minilm-l6-v2'. Did you mean: minilm-l6-v2?

FAQ

What does pip install kjarni pull in?

Nothing. The package declares no dependencies. numpy is used if it is already present, and encode_batch returns lists of lists when it is not.

Does it need a GPU?

No. Everything above runs on CPU. Pass device="gpu" to Indexer or Searcher to use one; the GPU path is Vulkan, Metal or DirectX 12, so AMD, Intel and Apple Silicon work as they are.

Which mode should I use?

Semantic when queries are phrased as questions or paraphrases. Keyword when they contain identifiers, product codes or names that must match exactly. Hybrid when they are a mix, with a reranker on top if precision at the first position matters.

Can I search an index built by another language?

Yes. The index format is the engine's, not the binding's, so an index written by the CLI or from C# opens unchanged in Python.

How large can an index get?

The store is on disk and searched without loading everything into memory. Chunk counts in the tens of thousands are ordinary on a laptop.

Is generation included?

Yes. kjarni.Chat("llama3.2-3b-instruct") runs a local LLM in the same process, with send for a whole reply and stream for token by token. The package also has Summarizer, Translator and Transcriber (Whisper), so the same pip install kjarni covers generation alongside search.

Getting it

pip install kjarni
PyPI:   https://pypi.org/project/kjarni/
GitHub: https://github.com/olafurjohannsson/kjarni
Docs:   https://kjarni.ai/reference/python/

Questions and bug reports go to GitHub issues.

Kjarni runs the same engine on every platform: