We all get excited when the big companies release their new frontier models — probably by now with trillions of parameters. And if you've been in the game for a little while, you see leaps and bounds with each new version. With this, we tend to think those large LLMs — Claude Opus, Gemini Ultra, GPT-4o — are the kings out there. And they probably are. But they also cost ya! You can't go too crazy using them, or you'll need a second mortgage to cover your AI bills.
For the last couple of weeks, I started playing with models that run locally on my machine. I was very impressed. Some small LLMs can really throw a punch. They're not at the level of the big guys — I haven't switched the LLM powering my coding tools yet. But for simpler tasks — summarizing text or transcripts, classifying inputs, generating content — those little guys can get the job done, and done well.
It shouldn't be such a surprise. A quick browse on Hugging Face and the insane number of options tells us a different story is unfolding. There are thousands of models, and they differ in ways that actually matter: how they've been compressed, how they've been fine-tuned, what format they're packaged in. With a little understanding of how they're organized, you can get a lot done without ever touching a paid API.
This article is about exactly that: making sense of the local LLM landscape — what the models actually are, how they're shrunk down to run on your hardware, and the tools to run them.
The Model Families You Need to Know
Before we get into the how, let's cover who's who. The model landscape splits into two camps: closed frontier models (API-only) and open-weight models (download and run yourself).
The closed-only crowd — OpenAI's GPT, Anthropic's Claude, Google's Gemini — these are the heavy hitters. They consistently top benchmarks, handle complex multi-step reasoning, and are the go-to for anything production-critical. The catch: you pay per token, your data goes to their servers, and you have no control over when the model changes under you.
The open-weight world is where things get interesting for local use. A few families dominate:
| Family | Origin | Best sizes for local | Known for |
|---|---|---|---|
| LLaMA 3.x | Meta | 3B, 8B, 70B | General-purpose benchmark king; started the open-weight wave |
| Mistral / Mixtral | Mistral AI (France) | 7B, 8×7B | Extremely efficient; Mixture-of-Experts gives big-model quality cheaply |
| Qwen 2.5 | Alibaba | 0.5B, 3B, 7B, 14B | Multilingual, strong at coding & math, tiny variants (0.5B!) work |
| Phi-3 / Phi-4 | Microsoft | 3.8B, 14B | Trained on curated "textbook" data — punches way above its size |
| Gemma 2 | Google DeepMind | 2B, 9B, 27B | Clean distillation from Gemini; excellent instruction following |
| DeepSeek V3 / R1 | DeepSeek | 7B, 14B | Frontier-class reasoning at open-weight cost; surprised everyone in 2025 |
Making Them Small: Quantization & Distillation
So how do you get a 70-billion-parameter model — which in its raw form needs around 140 GB of RAM — to run on your laptop? Two techniques: quantization and distillation. These aren't the same thing, and understanding the difference matters for picking the right model.
Quantization: Turning Down the Precision
A neural network is fundamentally a giant pile of numbers — billions of floating-point weights tuned during training. In their native form, each weight is stored as a 16-bit or 32-bit float. Quantization says: do we really need all that precision?
The answer, most of the time, is no. When you quantize a model to 4-bit integers (INT4), each weight takes up 4 bits instead of 16 — a 4× size reduction. An 8B model that would normally need ~16 GB of RAM now fits in roughly 4 GB. A 70B model that was GPU-only territory at 140 GB can now run on a 40 GB machine.
The trade-off is accuracy: you're rounding all those carefully tuned decimal values to a coarser number line. Lose too much precision and the model starts hallucinating more, reasoning worse, getting confused on edge cases.
The good news: modern quantization methods are remarkably good. The current k-quants approach (used in the GGUF format) applies different quantization levels to different layers of the model — keeping the most sensitive layers more precise. The rule of thumb that's held up well:
Distillation: Teaching a Small Model with a Big Teacher
Distillation is a completely different approach. Instead of compressing an existing model, you train a new, smaller model — the "student" — to mimic a larger, more capable model — the "teacher."
The teacher doesn't just hand over its training data. It hands over something richer: its probability distributions. When the teacher predicts the next word, it doesn't just say "cat" — it says "there's a 62% chance it's 'cat', 18% chance 'kitten', 8% chance 'dog'..." The student learns from these soft distributions, which carry far more information than simple right/wrong labels.
The result is a small model that behaves more like its large teacher than you'd expect from its size alone. Microsoft's Phi series is the canonical example — training small models on synthetically generated, textbook-quality data from larger models to get outsized reasoning performance. A 3.8B Phi-3 model regularly outperforms 7B models trained on raw web crawls.
Distillation gives you a new model rather than a compressed version. The quality floor is usually higher — you're not lossily compressing something, you're building something new, informed by a teacher. In practice, most capable local models are both distilled and quantized: trained with distillation techniques for quality, then quantized for distribution.
The Model File Zoo: What Are All These Formats?
Download a model from Hugging Face and you'll encounter file extensions that look like someone fell asleep on a keyboard: .gguf, .safetensors, .gptq, .awq. Here's what they actually mean.
GGUF: The Local Inference Standard
GGUF (GPT-Generated Unified Format) is what you want for running models locally on CPU or a consumer GPU. It was created by the llama.cpp team — a C++ inference engine that runs essentially everywhere.
GGUF files are self-contained: model weights, architecture metadata, tokenizer, and everything else needed to run live in a single file. You download one file, point your tool at it, done.
The quantization level is encoded right in the filename — once you learn to read it, it tells you everything:
Llama-3.1-8B-Instruct-Q4_K_M.gguf
│ │ │ │ │ │
│ │ │ │ │ └── size variant (M = medium calibration)
│ │ │ │ └────── k-quants quantization method
│ │ │ └─────────────── 4-bit quantization
│ │ └────────────────── Instruct = chat fine-tuned
│ └────────────────────── 8 billion parameters
└────────────────────────────── base model name
SafeTensors: The Training & Research Format
SafeTensors is Hugging Face's standard format for distributing models in their original precision. When you use the transformers library in Python, this is what you're loading. Full-precision (FP16 or BF16) — good for fine-tuning and research when you have a GPU with enough VRAM. Not typically what you want for CPU inference.
MLX: Apple Silicon's Native Format
If you're on an M-series Mac, MLX is your secret weapon. Apple Silicon has unified memory — CPU and GPU share the same RAM pool. This means a 16 GB MacBook can load a 16 GB model entirely "on GPU" in a way that's impossible on most laptops where GPU VRAM is separate and limited.
Models in MLX format run via Apple's Metal GPU at surprisingly high speeds — an M3 chip running a 7B model at 60–80 tokens per second feels genuinely real-time. The mlx-lm library handles both the format conversion and inference.
GPTQ and AWQ: GPU-Optimized Quantization
GPTQ and AWQ are quantization formats specifically designed for NVIDIA GPUs. AWQ (Activation-aware Weight Quantization) is newer and achieves slightly better quality at the same bit-width. If you have an RTX 3090 or better, these outperform GGUF on that hardware. Otherwise, stick with GGUF or MLX.
| Format | Best for | Hardware | Single file? |
|---|---|---|---|
| GGUF | Local inference, widest compatibility | CPU, GPU, Metal | ✅ Yes |
| SafeTensors | Fine-tuning, HuggingFace ecosystem | GPU (VRAM) | ❌ Multiple |
| MLX | Apple Silicon inference | M1/M2/M3/M4 | ✅ Yes |
| GPTQ | NVIDIA GPU inference | NVIDIA GPU | ✅ Yes |
| AWQ | NVIDIA GPU, better quality than GPTQ | NVIDIA GPU | ✅ Yes |
Finding the Right Model
With thousands of models and multiple formats, how do you find what you actually want?
Hugging Face Hub is the primary repository for the ML world. The GGUF model filter is your best starting point, sorted by downloads. Download count is a surprisingly good proxy for quality — the community gravitates quickly toward models that actually work.
Community quantizers like bartowski take official model releases and produce clean GGUF versions at every quantization level within hours of a new drop. When LLaMA or Qwen releases something new, their GGUF builds usually appear the same day.
Ollama's library (ollama.com/library) is a curated subset — fewer options, but everything is vetted and pulls with a single command. Think of it as the App Store version: less selection, zero setup headaches.
OpenRouter (openrouter.ai) is worth knowing about even if you're primarily running locally. It's a unified API for 100+ models across every provider — free-tier Llama and Mistral models are available. Great for comparing outputs before committing to running one locally.
Groq deserves a special mention: it's a cloud inference platform using custom hardware that runs Llama 3.1 8B at roughly 800 tokens/second. Not local, but nearly free and faster than reading — great for rapid experimentation.
Running Them: Your Options
You've picked a model. Now how do you actually run it? The good news: despite the apparent complexity, every tool ends up exposing an OpenAI-compatible chat API. Once you understand one, you understand them all.
Ollama: The Easiest Path
Ollama is by far the lowest-friction way to get a local LLM running. Install it, run one command, and you have a local model server in minutes. It automatically handles hardware detection — Metal on Mac, CUDA on NVIDIA, CPU everywhere else.
ollama pull llama3.2 # downloads the model (~2GB)
ollama run llama3.2 # interactive chat in terminal
ollama serve # starts REST API server on :11434
Ollama exposes an OpenAI-compatible endpoint, so any code you've written for the OpenAI API just works with a URL swap:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1",
api_key="ollama", # any string works
)
# Your existing OpenAI code works unchanged
response = client.chat.completions.create(
model="llama3.2",
messages=[{"role": "user", "content": "Hello!"}],
)
LM Studio: The Desktop GUI
LM Studio is a desktop application with a clean UI for downloading, managing, and chatting with local models. It pulls from Hugging Face directly, handles quantization selection for you, has a built-in chat interface, and runs a local OpenAI-compatible API server.
The key difference from Ollama is the interface: LM Studio is a GUI, making it approachable without command-line comfort. It also exposes fine-grained controls (temperature, context length, repetition penalty) visually. There's a small latency penalty from the server layer, but it's minor in practice. LM Studio is where many people start because it makes the whole ecosystem tangible.
llama.cpp and mlx-lm: The Power Tools
llama.cpp is the C++ inference engine that powers most of the local LLM ecosystem under the hood — Ollama itself wraps it. Running it directly via the Python wrapper llama-cpp-python gives you maximum control: custom GGUF files, advanced sampling parameters, precise GPU/CPU layer splitting:
from llama_cpp import Llama
llm = Llama(
model_path="./Llama-3.1-8B-Instruct-Q4_K_M.gguf",
n_gpu_layers=-1, # -1 = push everything to GPU
n_ctx=4096, # context window size
n_threads=8, # CPU threads for non-GPU layers
verbose=False,
)
output = llm.create_chat_completion(
messages=[{"role": "user", "content": "Explain attention in 2 sentences."}]
)
mlx-lm is the Apple Silicon equivalent. If you're on an M-series Mac and want maximum performance, this is your tool. The performance difference is substantial — a 7B model running at 20 tok/s via CPU-only llama.cpp will do 60–80 tok/s via mlx-lm on the same chip, using the Metal GPU and unified memory.
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Llama-3.2-8B-Instruct-4bit")
response = generate(model, tokenizer,
prompt="Explain attention.",
max_tokens=256, verbose=True) # shows tok/s
LiteLLM: One Interface for Everything
If you're building something that needs to work across multiple providers — sometimes local, sometimes cloud — LiteLLM is the answer. It's a Python library that gives every model the same OpenAI-compatible interface. You change one string and routing handles itself:
from litellm import completion
# All of these work identically:
completion(model="gpt-4o-mini", ...) # OpenAI
completion(model="claude-3-5-haiku-20241022", ...) # Anthropic
completion(model="ollama/llama3.2", ...) # Local
completion(model="groq/llama-3.1-8b-instant", ...) # Groq
Putting It Together: How to Pick
The decision tree is simpler than it looks once you've internalized the above:
The Bottom Line
The local LLM ecosystem has matured fast. Two years ago, running a capable language model locally meant serious tinkering. Today, it's a brew install ollama and a one-line model pull. The models themselves have improved dramatically — a Q4-quantized 8B model from 2025 outperforms what was only possible with 70B models two years ago.
None of this replaces the frontier models for hard tasks. But for a huge slice of real-world work — document summarizers, classifiers, code completers, text generators, translation tools, voice agents — the small, local models are good enough. They're free, fast, private, and getting better with every release cycle.
The kings are still the kings. But these days, the paupers are secretly pretty capable.
This Is Just the Beginning
What we've covered here — picking a model, understanding quantization, choosing a runtime — is really just getting you to the starting line. Once you have a local model running well for a task, a whole toolkit opens up for making it even better.
Fine-tuning is the natural next step. If your model is 80% of the way there but keeps getting your domain's terminology wrong, or isn't following the output format you need, a LoRA fine-tune on even a few hundred curated examples can close that gap dramatically. You're not retraining the whole model — just nudging a tiny fraction of its weights toward the behavior you want, using techniques like QLoRA that fit on consumer hardware.
RAG (Retrieval-Augmented Generation) is how you give a local model access to knowledge it wasn't trained on — your internal docs, a product catalog, a knowledge base — without retraining anything. The model fetches relevant chunks at query time and reasons over them. It's often the fastest path from a capable model to a useful product.
Agents and tool use let small models punch well above their weight. Give a 7B model a calculator, a web search tool, and a file reader, and suddenly it can complete multi-step tasks that would stump it in a single-shot setting. The model doesn't need to know all the answers — it just needs to know which tool to call.
The local LLM world moves fast. The model you run today may be outclassed by something half its size in six months. But the fundamentals — quantization, formats, runtimes, fine-tuning, RAG — those stay relevant. Once you understand the stack, keeping up with the new releases becomes a matter of swapping one model file for another.