How to Run a Local AI Model on Your Computer

What is Local AI?

"Local AI" means running a large language model (LLM) directly on your own computer rather than sending prompts to a cloud service like ChatGPT, Claude, or Gemini. The model weights are downloaded once -- typically a few gigabytes -- and then everything runs on your CPU or GPU.

There are three big reasons to care:

• •Privacy: Your prompts never leave your machine. Great for sensitive code, legal text, or personal notes.
• •Cost: No subscription fees, no per-token billing. Just electricity.
• •Offline: Works in a cabin, on a plane, during an outage -- wherever your laptop's battery lasts.

The two tools we'll use -- Ollama and Open WebUI -- are both free and open-source. Ollama is the engine that runs the model; Open WebUI is the ChatGPT-style chat interface on top of it.

Step 1. Check Your System Requirements

Modern quantized models can run on surprisingly modest hardware, but RAM is the big constraint since the whole model needs to fit in memory. Here's what to aim for:

If you're under 8 GB of RAM, stick to the smaller models and accept that responses will be slow. If you've got a MacBook with 16 GB+ unified memory, you're in great shape -- Apple Silicon is one of the best local AI setups out there, often beating similar-priced PC laptops thanks to high unified memory bandwidth.

You'll also need a terminal to run the commands. On Windows, use PowerShell or Windows Terminal. On macOS, the built-in Terminal app. On Linux, your distro's default terminal. Ollama also includes a small tray app on Windows and macOS so you don't always need the terminal.

Step 2. Install Ollama

Head to ollama.com and download the installer for your platform:

Once installed, verify it's working by checking the version:

Now download your first model. Llama 3.2 is a great default -- it's a 7B-class model from Meta, well-balanced on quality and speed. Pull it with:

The first download typically takes a few minutes depending on bandwidth -- Llama 3.2 is around 4 GB. Once it's down, the model is stored locally and you won't need to re-download it.

Step 3. Use Ollama from the Terminal

You can start chatting with the model from the terminal immediately. Run:

You'll land in an interactive prompt. Type a question and press Enter:

The model streams its response token-by-token. You can keep asking follow-ups -- the conversation context is preserved within the session. Useful in-chat commands:

• •/bye -- Exit the chat session.
• •/clear -- Wipe the conversation context to start fresh.
• •/show info -- Display model details (parameters, quantization, size).
• •/? -- List all available slash commands.

You can also pipe single prompts non-interactively. Great for scripting:

Step 4. Install Open WebUI for a ChatGPT-style Interface

The terminal is fine for quick tests, but for a proper chat experience you want a real interface. Open WebUI is an open-source web app that gives you ChatGPT-style conversations, model switching, document uploads, and a clean history -- all running locally against your Ollama install.

Prerequisites: you need Docker installed. On Windows/Mac, Docker Desktop is the easiest path. On Linux, install Docker Engine from your distro's repos.

Once Docker is ready, pull and launch Open WebUI with this single command:

What this command does:

• •-d -- Run in detached mode so it keeps running in the background.
• •-p 3000:8080 -- Map host port 3000 to the container's 8080.
• •--add-host=host.docker.internal:host-gateway -- Lets the container reach Ollama running on your host machine (not inside the container).
• •-v open-webui:... -- Persist chat history, settings, and user data in a named volume.
• •--restart always -- Auto-restart the container on reboot or crash.

The first launch pulls the image (~1 GB) and starts Open WebUI in the background. You can follow logs with:

Step 5. Access Open WebUI and Chat Locally

Once the container is running, point your browser at:

1. 1.On first visit, Open WebUI will ask you to create an admin account. Pick any username and password -- it's stored locally only.
2. 2.After signup you'll see the chat interface. Top-left has a model dropdown -- it should auto-discover llama3.2 from your Ollama install.
3. 3.Select a model, type a question, hit Enter. The reply streams back just like ChatGPT -- but it's running entirely on your machine.
4. 4.You can create separate chats, name them, and they persist across restarts thanks to the Docker volume.
5. 5.Open WebUI also supports document uploads -- RAG over your own PDFs and text files using a local embedding model.

To pull more models through the UI, use the Settings > Models page, or pull them from the terminal with ollama pull <model> and they'll show up automatically.

Popular Models to Try

• •llama3.2 -- Meta's all-rounder. Good general chat, code, summarizing. ~4 GB.
• •mistral -- Mistral AI's 7B, fast and well-balanced. Good at coding and reasoning. ~4 GB.
• •phi3 -- Microsoft's tiny but capable 3.8B model. Great if you're RAM-constrained. ~2 GB.
• •gemma2 -- Google's lightweight line. The 2B version is excellent for very small machines. ~1.5-5 GB depending on variant.

Try them, compare them, and see which one fits your style. There's no cost to switching or keeping several installed.