Pick a local LLM that runs on your hardware
Most guides assume you have an RTX 4090 and unlimited patience. This one meets you where you are — Mac, budget PC, or beefy workstation — and tells you which model actually runs well on it.
Updated September 2026 · Open-source · Free to read
Watch the deep dive
Episode 4 of the AI Beginner Journey — a 30-minute video walk through cloud vs local, model size, quantisation, and exactly which models fit each hardware tier.
Watch on YouTube →Why pick by hardware, not by leaderboard
A 70B model will outperform a 7B model on every leaderboard. But if your machine can't load it, that's irrelevant. The right model is the one that runs smoothly on the hardware you have— not the one that wins benchmarks your laptop can't touch.
For everyday tasks (email drafting, summarisation, code snippets, brainstorming), the gap between a well-quantised 14B model on a 4090 and a 70B on a server is smaller than you'd think. For research and long-context work, the gap grows. Pick the model that matches the task and the machine.
Start here — what's your hardware?
RTX 3080 (10 GB)
Small model tier10 GB of VRAM is plenty for everyday AI. You'll be running 7B to 9B parameter models in Q4 quantisation — fast, surprisingly capable, and good for chat, drafting, summarisation, basic code.
🥇 First pick — Qwen3.5 7B (Q4_K_M)
Strong general model, good at code, runs at ~60 tokens/sec on a 3080. Best default.
ollama pull qwen3.5:7b-instruct-q4_K_M🥈 Alternate — Gemma4 9B (Q4_K_M)
Google's open-source model, slightly larger context window, edge on creative writing.
ollama pull gemma4:9b-instruct-q4_K_M🥉 Alternate — Mistral 7B (Q4_K_M)
Efficient, often best-in-class per FLOP. Falls back gracefully on hard prompts.
ollama pull mistral:7b-instruct-q4_K_MWhat to skip: anything 30B or bigger. It will technically load via CPU offloading but feel like 2003 dial-up.
RTX 4090 (24 GB)
⭐ Best value tier24 GB of VRAM is the sweet spot. You can run 30B-class models in good quantisation — quality comparable to mid-tier cloud APIs, fully local, free.
🥇 First pick — Qwen3-coder 30B (Q4_K_M)
Strongest open-source coding model on this hardware. Excellent for code review, refactoring, scripts, debugging.
ollama pull qwen3-coder:30b-instruct-q4_K_M🥈 Alternate — DeepSeek R1 32B (Q4_K_M)
Best reasoning in this size class. Slower than Qwen-coder but stronger on logic and planning.
ollama pull deepseek-r1:32b-q4_K_M🥉 Alternate — Gemma4 26B (Q4_K_M)
Google's newest. Solid general-purpose, good at writing, balanced for chat.
ollama pull gemma4:26b-instruct-q4_K_MBonus: the 4090 from 2023 is still excellent for this work. If you have one, you're set — no upgrade required.
Mac M-series (32-64 GB unified)
Unified memory tierApple Silicon Macs use the same memory pool for CPU and GPU. A 32 GB Mac runs models that need 32 GB of VRAM on a PC — no separate GPU required. The cheat code of local LLM.
🥇 Mac with 32 GB unified — Qwen3 27B (Q4_K_M)
Best mid-size general model for this hardware. Fast, balanced, great for chat and coding.
ollama pull qwen3:27b-instruct-q4_K_M🥇 Mac with 64 GB unified — Llama 3.3 70B (Q4)
Cloud-quality reasoning locally. If you have 64 GB, this is the ceiling.
ollama pull llama3.3:70b-instruct-q4_K_M📌 Mac-specific tip
Use ollama with the Metal backend — Ollama auto-detects Apple Silicon. LM Studio works too with the same models. Don't bother with llama.cpp directly unless you want maximum control.
If you already own a Mac with 32+ GB unified: you're set. Don't buy new hardware for local LLM. The cost-per-output already beat any cloud subscription.
NVIDIA DGX Spark (128 GB unified)
Workstation tierNVIDIA's small desktop supercomputer. 128 GB of unified GB300 memory runs 70B and 120B parameter models locally, at workstation scale.
🥇 First pick — Llama 3.3 70B (FP16)
Full precision 70B. Maximum quality at this hardware level.
ollama pull llama3.3:70b-instruct🥈 Alternate — Qwen3 120B (Q4_K_M)
Cloud-API-class reasoning, fully local, with budget to spare for context.
ollama pull qwen3:120b-instruct-q4_K_M🥉 Alternate — DeepSeek R1 70B (Q4_K_M)
Top-tier open reasoning model. Excellent for planning, math, code.
ollama pull deepseek-r1:70b-q4_K_MReal talk: DGX Spark is overkill for personal use. It's priced for teams and small businesses that want GPT-4-class reasoning on their own infrastructure. If you're not running one already, the RTX 4090 covers 90% of use cases for 5% of the cost.
Quantisation primer — Q4_K_M and friends
When you see "Q4_K_M" in a model name, that's the quantisation level. The original model uses 16-bit numbers. Quantisation converts them to 8-bit, 4-bit, or even 2-bit. The file gets smaller. Quality drops a little. Speed usually goes up.
| Quant | Size vs original | Quality loss | When to use |
|---|---|---|---|
| Q2 | ~8× smaller | Noticeable | Last resort only |
| Q4_K_M | ~4× smaller | ~1-2% | The default. Best balance. |
| Q8 | ~2× smaller | Negligible | Quality mode, spare VRAM |
| FP16 (original) | 1× | None | Best quality, hardest to run |
Rule of thumb: if unsure, pick Q4_K_M. Quality loss is small, model fits most hardware, speed is good.
How to install — three steps
Install Ollama
Free. Mac, Windows, Linux. ~5 minutes.
curl -fsSL https://ollama.com/install.sh | shOr download from ollama.com if you prefer a GUI installer.
Pull a model
Pick a model from the list above for your hardware. Downloads take a few minutes depending on your internet.
ollama pull qwen3.5:7b-instruct-q4_K_MRun it
Ask a question. Get an answer. You're now running a local LLM.
ollama run qwen3.5:7b-instruct-q4_K_M "Hello, what can you help me with?"Prefer a GUI? LM Studio is the same idea with a clicky-clicky interface.
The model families — who's who
Open-source LLM ecosystem in 2026. You don't need to memorise all of these — pick one good family, learn its quirks, switch later if needed.
Quick answers
What's the difference between 7B, 14B, 27B, 70B?
The B = billion parameters. More parameters = more nuance the model has learned, but also more memory needed and slower speeds. 7B fits anywhere; 27B needs a 4090 or 32 GB Mac; 70B needs 64 GB+ or a workstation.
Is local LLM actually private?
Yes — your prompts never leave your machine. No logs, no telemetry, no server audit. The only network traffic during inference is whatever your machine already does (typically nothing).
Why doesn't a 70B model beat a 7B model on every task?
It usually does — but for everyday tasks (email drafts, summarisation, basic code), the gap is small. You can drop $0/month on a 7B local model and never notice. The gap widens on hard reasoning, long contexts, and novel situations.
How fast is "fast enough"?
Aiming for 30 tokens/sec is realistic and "feels right." Below that you'll start waiting; above that you won't notice the difference. If your model is at 8 t/s, drop quantisation to Q4_K_M first.
Should I buy a 4090 or use my existing Mac?
Existing Mac. Always. 64 GB unified on a Mac Studio beats a 4090 on cost-per-output and runs the same Qwen/DeepSeek models. The 4090 wins if you need CUDA-specific tools or higher token/sec on the largest models.
Ready to try local AI tonight?
Pick the model for your hardware above. Run the three install steps. Ask your first question in 20 minutes.
Free · No signup · Runs on your hardware
From the AI Beginner Journey series on YouTube. ClawPack home.