Choose the Right Local AI Model

What Are Local AI Models?

Local AI models are large language models (LLMs) that run entirely on your own computer — no cloud, no subscriptions, no data leaving your machine. They give you complete privacy and work offline. You run them through tools like Ollama, which handles downloading, running, and managing models with a single command.

But not all models are created equal. They come in different sizes (measured in parameters), are built by different companies with different strengths, and have wildly different hardware requirements. This guide helps you pick the right one for your setup.

• •Some models are tiny and fast — perfect for older laptops or quick tasks
• •Others are massive and brilliant — but they need serious hardware
• •There are specialist models for coding, multilingual conversation, and vision

Step 1. Understand the Model Families

Each major AI research group releases models under its own banner. Knowing who makes what helps you understand the personality and strengths of each option.

• •Meta (Llama 3.2 / 3.3) — The most popular open-weight models. Balanced, well-rounded, and great all-rounders. Llama 3.2 3B is excellent for everyday tasks on modest hardware.
• •Mistral AI — French-made models known for strong reasoning and instruction-following. Mistral 7B punches above its weight class, and Mixtral 8x7B uses a clever "mixture of experts" design to run efficiently.
• •Microsoft (Phi-4) — Compact but surprisingly capable. Phi-4 Mini (3.8B) rivals much larger models on reasoning and math. Ideal when you want smarts in a small package.
• •Google (Gemma 2) — Google's open models. Gemma 2 9B is a standout for its size — fast, accurate, and great at following instructions. The 27B version competes with much larger models.
• •DeepSeek — Chinese models that excel at coding and math. DeepSeek-R1 Distill variants bring strong reasoning into smaller sizes (7B, 8B, 14B). The full R1 is enormous but the distilled versions are very usable locally.
• •Qwen 2.5 (Alibaba) — Underrated workhorses. Qwen 2.5 models (0.5B through 72B) are excellent at code and multilingual tasks. The 7B and 14B versions are frequently overlooked gems.

Step 2. Check Your RAM — The Hard Limit

The single most important factor when choosing a local model is your RAM. Models are measured in parameters — think of each parameter as a dial the model can tune. More parameters = more knowledge and reasoning, but also more memory.

Here's a rough guide to how much RAM you need at different parameter counts (assuming 4-bit quantisation, which is what Ollama uses by default):

• •1B–3B parameters — 2–4 GB RAM. Any computer made in the last decade can run these. Perfect for Raspberry Pi 5, old laptops, or Chromebooks.
• •7B–8B parameters — 6–8 GB RAM. The sweet spot for most people. Runs smoothly on M-series Macs with 8 GB and most Windows/Linux laptops with 16 GB.
• •13B–14B parameters — 10–12 GB RAM. A meaningful step up in quality. Needs a Mac with 16 GB or a gaming PC with a dedicated GPU.
• •27B–33B parameters — 20–24 GB RAM. These models are genuinely smart. You'll want a Mac with 24–32 GB or a PC with a modern GPU (RTX 3060+).
• •70B+ parameters — 40+ GB RAM. The big leagues. Requires a high-end workstation, Apple Silicon Mac with 64+ GB unified memory, or a multi-GPU setup.
• •100B–200B parameters — 60–120 GB RAM. Entering datacentre territory. Models this size (like Llama 3.1 405B, DeepSeek V3 671B) are technically open-weight but practically impossible to run on consumer hardware. Even with multi-GPU servers, you're looking at thousands in hardware — this is cloud-only territory for nearly everyone.

Not sure how much RAM you have? Here's how to check:

Step 3. Understand the Speed vs. Quality Tradeoff

Smaller models are faster but less capable. Larger models think harder but take longer. You need to decide where you sit on that spectrum for each task.

• •Quick chat, grammar check, simple Q&A — 1B–3B models are instant and perfectly fine for this. Try Llama 3.2 3B or Phi-4 Mini.
• •Daily assistant, writing help, research — 7B–8B is the daily-driver tier. Llama 3.2 7B, Mistral 7B, or Gemma 2 9B will handle most tasks with aplomb.
• •Serious analysis, long-form writing, complex reasoning — 13B–14B. Qwen 2.5 14B or DeepSeek-R1 Distill 14B deliver noticeably sharper answers.
• •Professional work, creative projects, heavy brainstorming — 27B–33B. Gemma 2 27B or Qwen 2.5 32B are genuinely impressive and approach cloud-model quality.
• •Maximum local intelligence — 70B+. Llama 3.3 70B is the current king of open models, but you pay for it in hardware.

A good strategy: keep two models installed. A fast 3B for quick questions and a smarter 8B–14B for serious work. Ollama makes switching as easy as typing a different model name.

Step 4. Pick Specialised Models for Specialised Tasks

General-purpose models are great, but sometimes you need a specialist. These fine-tuned models outperform larger general models on their specific domain.

• •Code generation — CodeGemma 7B, DeepSeek-Coder V2, or Qwen 2.5 Coder 7B. These are trained specifically on code and will produce better, more idiomatic results than general models of equal size.
• •Math and logic — DeepSeek-R1 Distill variants (7B, 8B, 14B). These use "chain of thought" reasoning — the model thinks before answering, dramatically improving accuracy on hard problems.
• •Multilingual tasks — Qwen 2.5 and Llama 3.2 offer strong multilingual support across dozens of languages. If you regularly work in non-English languages, prioritise these.
• •Vision / image understanding — Llama 3.2 Vision (11B, 90B) and Llava can look at images and describe them, read text from screenshots, or analyse photos. Requires more RAM than text-only models of the same size.
• •Long documents — Some models support huge context windows (128K+ tokens — enough for a whole book). Llama 3.2, Mistral, and Qwen 2.5 all offer extended-context versions. Great for summarising long PDFs or analysing legal documents.

Step 5. Frontier Models — What You Can't Run (Yet)

The AI world moves fast, and the most capable models are often the largest. Several recent releases push well beyond what any consumer device can handle — they're interesting to know about, but not candidates for your local setup.

GLM-5.2 (Zhipu AI / Tsinghua)

A Chinese bilingual (Chinese + English) model with exceptional reasoning and tool-use capabilities. GLM-5.2 scored near the top of the Chatbot Arena leaderboard, competing head-to-head with GPT-4o and Claude 3.5. It's a massive dense model — way beyond consumer GPUs. If you encounter it, it's running on Zhipu's cloud or a datacentre cluster, not under your desk.

Kimi K3 (Moonshot AI)

Kimi K3 is a reasoning-focused model from Moonshot AI that uses an extremely long context window (up to 1 million tokens) combined with chain-of-thought reasoning. It's a mixture-of-experts architecture with hundreds of billions of parameters total — the kind of model that needs multiple H100 GPUs working together. Excellent at long-document analysis and multi-step research tasks, but firmly in the cloud-only category.

Fable 5 (Fable AI)

Fable 5 is a creative-writing specialist — it produces remarkably natural dialogue, narrative prose, and character-driven storytelling. It's a large proprietary model that hasn't released open weights. If creative writing is your main use case and Fable's style appeals to you, you'd need to use it through their cloud API — it won't be coming to Ollama.

Why Frontier Models Stay in the Cloud

These models share a few traits that keep them out of consumer reach:

• •Sheer size. Dense models over 100B parameters need multiple high-end GPUs just to load into memory. A single H100 GPU costs £20,000+ — and you'd need several.
• •Mixture-of-experts architectures. Even when total parameters are huge, the active parameters per query are smaller — but the full model still must sit in RAM across multiple GPUs.
• •Closed weights. Many frontier models (Fable, Kimi in some versions, GLM in some versions) don't release their model weights at all — they're only accessible through paid APIs.
• •Infrastructure requirements. Even if you could load the model, serving it at usable speed requires high-bandwidth GPU interconnects (NVLink, InfiniBand) that don't exist in consumer hardware.

The good news: techniques like quantisation, distillation, and model merging are steadily shrinking frontier capability into consumer-friendly sizes. DeepSeek-R1 Distill 8B and 14B already bring some of R1's reasoning into compact packages. The gap between \"frontier\" and \"local\" closes a little every quarter. For now, pick the best model that fits your hardware — you'll be surprised how capable 7B–14B models have become.

Step 6. Test and Compare Models Side-by-Side

The best way to choose is to try. Ollama makes it trivial to download and run multiple models. Pull 2–3 candidates and ask them all the same questions — you'll quickly develop a feel for which one fits your style.

When comparing, pay attention to:

• •Speed — How fast does the first token appear? How fast are subsequent tokens? A model that takes 10 seconds to start typing is frustrating for chat.
• •Accuracy — Does the answer actually address what you asked? Does it hallucinate facts?
• •Tone — Some models are verbose, others concise. Some are formal, others casual. Find one whose voice you enjoy reading.
• •Instruction following — When you say "be brief" or "use bullet points," does it actually listen?

Step 7. Our Recommendation Matrix

Still not sure? Here's our cheat sheet for the most common scenarios:

These are starting points. Once you're comfortable, experiment! One of the joys of local AI is trying new models as they release — the landscape improves every month.

Quick Tips

• •Use ollama list to see what models you already have downloaded and how much disk space they're using.
• •Models are quantized by default in Ollama — you don't need to do anything special. A 7B model takes about 4–5 GB of disk space.
• •You can run multiple models simultaneously if you have enough RAM. Your AI agents can even use different models for different tasks.
• •New models drop every few weeks. The Ollama library and Hugging Face are the best places to discover fresh releases.
• •If a model feels slow, try the next size down. The difference between a 7B and an 8B is usually negligible in quality but sometimes the 7B fits in memory better.