LLMs on a CPU: QLoRA, GGUF and llama.cpp

Not every project can afford a GPU server, and a surprising number of useful LLM workloads don't need one. A large part of my work is making small models genuinely useful on ordinary hardware — fine-tuning models like Phi-4-mini with QLoRA, then quantizing to GGUF and serving with llama.cpp on a machine with no GPU in sight.

The workflow, end to end

Fine-tune with 4-bit QLoRA so training fits on modest hardware: the base model stays frozen and quantized while small adapter matrices absorb the task-specific learning. For narrow tasks — a text-to-SQL model, a classifier, a structured extractor — a few thousand good examples on a 3–4B model goes a long way.

After training, merge the adapters into the base model, convert to GGUF, and quantize. Q4_K_M is usually the sweet spot: quality loss over Q8 is barely measurable on narrow tasks, and the speed and memory win is large. Going below Q4 starts to hurt in ways you can actually feel in the outputs.

What the numbers look like

A 3.8B model at Q4_K_M runs at usable interactive speeds on a modern laptop CPU — a few tokens a second faster than reading speed for short generations, with a memory footprint that fits comfortably alongside the rest of an application. That's the regime where a local LLM stops being a demo and becomes infrastructure.

Where this shines

Edge4-SQL, my text-to-SQL project, runs a 3.8B model fully on CPU this way. For narrow, well-defined tasks, a small fine-tuned model on a CPU beats calling a giant API — cheaper, private, and fast enough. The query grammar is fixed, the schema vocabulary is small, and the failure modes are checkable: either the SQL parses and runs, or it doesn't.

The same logic applies to classification, extraction and routing. Anywhere the output space is small and the input style is consistent, the small model's weakness — world knowledge, open-ended reasoning — barely matters.

Where it doesn't

Be honest about the boundary. Long, multi-step reasoning, broad domain knowledge and nuanced writing are still the big models' game, and no amount of quantization closes that gap. My rule: if the task can be checked programmatically, a small local model can own it; if success depends on open-ended judgment, call the big model.

Practical advice if you're starting

Spend your data budget on the input format, not the volume — consistent prompts teach faster than more examples in inconsistent formats. Benchmark on the task, not on perplexity; a model can lose perplexity and win the task. And keep the unquantized model around: every quantization question is easier to answer when you can compare against the source.