· 7 min read

Local LLM Chat in C# without Ollama or an API Key

Run Llama, Mistral, Qwen or Phi locally from C#. One NuGet package, loaded in-process and answering offline. Streaming, multi-turn conversations and sampling control in .NET.

Running a language model from .NET usually means one of two things: an HTTP call to someone else's GPU, or installing Ollama and talking to a daemon on localhost:11434.

Kjarni is a NuGet package. The model runs inside your process.

using Kjarni;

using var chat = new Chat("llama3.2-3b-instruct", quiet: true);
Console.WriteLine(chat.Send("Explain retrieval-augmented generation in one sentence."));

The weights download once to a local cache, and everything after that is offline.

Install

dotnet add package Kjarni

Native binaries for linux-x64, linux-arm64, win-x64 and osx-arm64 ship inside the package and are picked automatically by RID. The only runtime dependency is glibc.

Streaming

Waiting for a complete response feels broken. Stream tokens as they are produced:

chat.Stream("Write a haiku about Reykjavík.", token =>
{
    Console.Write(token);
    return true;      // return false to stop generating
});

The callback returns a bool, which is also your cancellation mechanism. Return false and generation stops immediately rather than running to completion and throwing the result away.

Multi-turn conversations

A Chat is stateless between calls. For a conversation that remembers, ask it for one:

var convo = chat.Conversation();

convo.Send("My name is Ólafur.");
Console.WriteLine(convo.Send("What is my name?"));
// Your name is Ólafur.

Console.WriteLine(convo.Length);   // 4: two user messages, two replies
convo.Clear();                     // keeps the system prompt

ChatConversation also streams, via the same Stream(message, onToken) shape, and appends both sides to the history for you.

System prompts and modes

using var chat = new Chat(
    "llama3.2-3b-instruct",
    systemPrompt: "You are a terse assistant. Answer in one sentence.",
    mode: ChatMode.Default,      // or Creative, Reasoning
    quiet: true);

Controlling the output

var config = GenerationConfig.Default() with
{
    Temperature = 0.2f,
    MaxNewTokens = 512,
};

chat.Send("Summarise this changelog.", config);

GenerationConfig exposes Temperature, TopK, TopP, MinP, RepetitionPenalty, MaxNewTokens and DoSample, with three presets:

PresetUse it for
GenerationConfig.Default()model defaults
GenerationConfig.Greedy()deterministic output, Temperature = 0
GenerationConfig.Creative()higher temperature, more variation

Reach for Greedy() when you are doing extraction or classification with an LLM and want the same answer every time.

Which model?

Run kjarni model list for the full registry. The practical range:

ModelGood for
qwen2.5-0.5b-instructsmallest; simple structured tasks
llama3.2-1b-instructfast on modest hardware
llama3.2-3b-instructthe sweet spot on CPU
phi3.5-minireasoning-leaning, 3.8B
mistral-7bstronger, needs more RAM
deepseek-r1-8breasoning, largest

Real numbers for llama3.2-3b-instruct, on a desktop i7-13700 with an RTX A2000. Decode only, so model load and prompt processing are excluded, best of several interleaved runs on an otherwise idle machine:

WeightsDevicekjarniPyTorchllama.cpp
bfloat16CPU3.82.4n/a
bfloat16GPU25.4n/an/a
Q4_K_MCPU6.8n/a7.9
Q4_K_MGPU46.4n/a68.5

Bold marks the fastest where there is something to compare against.

The bfloat16 CPU row was measured in October 2026 and the rest in November. The gaps marked n/a are not omissions: PyTorch does not read GGUF files, llama.cpp does not read bfloat16 safetensors, and the PyTorch build here is CPU-only. Every engine within a row reads the identical file, so the rows are comparisons and the columns are not.

Model load is a few seconds from a warm page cache, and the context window is 131,072 tokens.

Quantisation is the single biggest lever, and it is worth understanding why. Generating one token reads every weight in the model exactly once, so decode speed is set by how fast weights stream out of memory rather than by arithmetic. Four-bit weights are a quarter the size of bfloat16, so they arrive four times faster.

Kjarni is about 1.6x PyTorch on the same weights, which is the number that matters if you are moving off a Python service. Against llama.cpp on the same quantised file it is about 14% behind on CPU. On GPU llama.cpp is about 1.5x faster over Vulkan, and further ahead with CUDA. It has years of hand-tuned kernels behind it and kjarni's GPU path is newer. If raw throughput is what you are optimising for, use llama.cpp.

If you want to check any of this yourself:

kjarni generate "The capital of Iceland is" \
  --model llama3.2-3b-instruct --model-path model.gguf \
  --max-tokens 100 --temperature 0 --repetition-penalty 1.0

The command line applies a repetition penalty of 1.1 unless told otherwise, so --repetition-penalty 1.0 is what makes this plain greedy decoding, the same as llama.cpp's defaults.

That 128K context is not a typo. It is the model's native window, and you are not paying per token to use it.

What this is not

The limits, plainly.

No tool calling yet. The engine supports it in principle; the C# surface does not expose it. If you need function calling, this is not your library today.

The GPU path is newer than the CPU path. Rather than ask you to take that on trust, here is the same prompt, the same Q4_K_M file, greedy decoding, on both devices and against llama.cpp reading the identical file:

kjarni CPU  Reykjavik, which is also the largest city in the country. Reykjavik is a
            popular tourist destination known for its vibrant cultural scene, stunning
kjarni GPU  Reykjavik, which is also the largest city in the country. Reykjavik is a
            popular tourist destination known for its vibrant cultural scene, historic
llama.cpp   Reykjavik, which is also the largest city in the country. Reykjavik is a
            popular tourist destination known for its vibrant cultural scene, historic

All three agree for the first twenty-odd tokens. Kjarni's GPU output then stays with llama.cpp for another sentence and a half before the two part ways.

That parting is normal for quantised weights, and it will happen with any two engines. Each one dequantises the four-bit blocks and sums the products in its own order, so the scores differ in the last few bits. Greedy decoding picks the single highest-scoring token, and when two candidates score within that rounding of each other, the difference decides the word and every token afterwards is conditioned on it. Between quantised engines, matching text token for token is not the right test. Producing an equally sensible continuation is.

At full precision the test is fair, and Kjarni passes it. On the original bfloat16 weights its greedy output matches Hugging Face transformers token for token: on this prompt for all 100 tokens, and on every chat prompt we test with across Qwen 2.5 0.5B and 1.5B, Llama 3.2 1B and 3B, Phi-3.5, Mistral 7B and DeepSeek R1 8B.

It is not going to beat llama.cpp on GPU throughput. That is not the trade. The trade is that chat, embeddings, classification, reranking and search all come from one dependency-free package you already know how to install.

FAQ

Does this need Ollama installed?

No. There is no daemon, no server, and no localhost port. The model runs in your process.

Does it need Python or PyTorch?

No. The NuGet package ships a native library. It runs inside your process, and the GPU path uses Vulkan, Metal or DirectX 12, so AMD, Intel and Apple Silicon work as they are.

Does it need a GPU?

No. Every code sample above runs on CPU, and the CPU numbers in the table are the ones to plan around. A GPU makes decoding nearly five times faster on bfloat16 weights and two and a half times faster on Q4_K_M, but nothing here requires it and there is no CUDA toolkit to install either way.

Where do the model weights go?

A local cache on first use, reused afterwards. Pre-download with kjarni model download llama3.2-3b-instruct to avoid a long pause on the first call.

Note that Llama models live in gated Hugging Face repositories, so downloading them needs an accepted licence and an HF_TOKEN in your environment. Qwen and Phi are openly downloadable and need no token.

Can I use it in a Docker container?

Yes. A plain .NET runtime image is enough: the library links only against glibc, and there is no separate model server to run beside it.

Does it implement IChatClient from Microsoft.Extensions.AI?

Yes, in Kjarni.Extensions.AI:

using IChatClient client = new KjarniChatClient("llama3.2-3b-instruct");
var response = await client.GetResponseAsync("Explain RAG (retrieval-augmented generation) in one sentence.");

That gives you streaming, multi-turn history and dependency injection through the standard abstraction, so Kjarni can stand in for any other IChatClient, including inside Semantic Kernel. The Chat API above is Kjarni's own, and is the lower-level option if you don't want the abstraction.

What does it cost?

Nothing. MIT or Apache-2.0 licensed, no API key, no per-token billing.

Try it

dotnet add package Kjarni

Source and issues: github.com/olafurjohannsson/kjarni

Kjarni runs the same engine on every platform: