Why run AI on your own machine
Three reasons, in the order people usually discover them.
Privacy. With a cloud service, everything you type and every document you attach travels to someone else's servers. For casual questions that is fine. For client work, source code, contracts, medical notes or anything covered by a confidentiality obligation, it is a genuine problem — and one that no privacy policy fully solves, because you are trusting a promise rather than an architecture. A local model removes the question entirely: there is no request to log because there is no request.
Cost. No subscription, no per-token billing, no usage tier to outgrow. Once the model is downloaded, running it a thousand times costs the same as running it once.
Independence. No rate limits, no outages, no model being deprecated out from under a workflow you built on it, and no internet connection required. The model you downloaded keeps behaving exactly the same way next year.
The counterweight is capability, and it is real. Set expectations correctly and local AI is excellent; expect it to match a frontier cloud model and you will be disappointed within an hour.
What your hardware can actually run
This is the part that determines everything else, so start here rather than with model names.
Language models are described by parameter count — 3B, 8B, 70B and so on — and the model must fit in memory to run at a usable speed. On a Mac with Apple Silicon, memory is shared between CPU and GPU, so total RAM is the figure that matters. On a Windows PC with a dedicated graphics card, VRAM matters most, though models can spill into system RAM at a significant speed cost.
- 8 GB RAM. Small models, roughly 3–4 billion parameters. Fine for summarising, rewriting and simple questions. It will struggle with anything requiring multi-step reasoning, and you will feel the limits fairly quickly.
- 16 GB RAM. The sweet spot. 7–8 billion parameter models run comfortably and are genuinely useful for daily work. This is the configuration where most people stop describing local AI as a toy.
- 32 GB RAM or more. Larger models become viable, and the quality gap against cloud services narrows considerably. Response times stay reasonable on Apple Silicon or a modern GPU.
- A dedicated GPU. Not required, but transformative for speed. The difference between waiting for a reply and watching it appear is mostly a GPU question.
Rule of thumb: a model needs roughly its parameter count in gigabytes, at common quantisation levels — an 8B model wants about 8 GB free, on top of whatever your operating system is already using. If you have 16 GB total and 10 GB free, an 8B model fits; a 13B model will not, and it will crawl.
Setting it up
Two pieces: something to run the model, and something to talk to it with.
1 Install Ollama
Ollama handles downloading models, loading them into memory and serving them to other applications. It runs in the background and needs no configuration. There are native installers for both Windows and macOS — on Windows, WSL2 is no longer required.
2 Pull a model that fits your RAM
From a terminal:
ollama pull llama3.2Then check what you have:
ollama listPick by the sizing guide above rather than by which model was in the news. Model releases move fast, so browse Ollama's library for what is current — the sizing logic stays the same even as the names change. The download runs to several gigabytes and happens once.
3 Install a real interface
Ollama on its own gives you a terminal prompt, which is fine for testing and tedious for daily use. Install the free OpenClaw Desktop App on the same machine for a proper chat interface, access to your files, scheduled tasks, and the option to reach the model from WhatsApp or Telegram on your phone.
In the app's provider settings, choose Ollama and select the model you pulled. It detects local models automatically.
4 Prove it is really local
Send a message and confirm you get a reply. Then turn off your Wi-Fi and send another one. If it still answers, everything is running on your hardware and nothing is going anywhere.
What local models are good and bad at
Being specific here saves a lot of disappointment.
Genuinely good
- Summarising. Long documents, meeting notes, articles. Local models handle this well and it is the most common daily use.
- Rewriting and drafting. Tone changes, shortening, turning bullet points into prose.
- Extraction. Pulling dates, names, figures and action items out of unstructured text.
- Questions about your own files. Where privacy matters most, and where local models are perfectly adequate.
- Classification and routing. Sorting, tagging, triaging — high volume, low complexity, exactly the profile that suits a small model.
Genuinely weak
- Complex multi-step reasoning. This is the clearest gap against frontier cloud models, and it does not close with prompting tricks.
- Long agentic chains. Tasks needing ten dependent steps tend to drift off course.
- Niche factual knowledge. Smaller models know less and, worse, are less reliable about admitting it.
- Long context. Local models have smaller practical context windows, so very large documents need chunking.
- Serious coding work. Fine for short snippets; not a substitute for a frontier model on real engineering tasks.
The hybrid approach most people settle on
Framing this as local-versus-cloud is a false choice. In practice the sensible arrangement is to route by sensitivity and difficulty.
Anything touching client data, source code, financial or personal records goes to the local model, because that is where privacy is non-negotiable and the task is usually summarising or extraction — work local models do well. Hard reasoning, research and long agentic tasks on non-sensitive material go to a cloud model, because that is where the capability gap actually bites.
The OpenClaw Desktop App lets you switch providers without rebuilding anything, so this stays a per-task decision rather than a permanent commitment. Many people run a local model as the default and reach for a cloud model when something genuinely needs it — which turns out to be less often than expected.
Troubleshooting
Responses are extremely slow
The model does not fit in available memory and is spilling to disk. Drop to a smaller model, or close whatever else is consuming RAM. This is by far the most common complaint and it is nearly always a sizing problem rather than a broken install.
The model gives short or unhelpful answers
Small models are literal. They respond much better to explicit instructions than to conversational hints — "summarise this in five bullet points, focusing on decisions and deadlines" will outperform "what do you think of this?" by a wide margin.
Out-of-memory errors
Straightforwardly too large a model for the machine. Move down one size tier.
The desktop app cannot see the model
Confirm Ollama is actually running — it is a background service and can be stopped without you noticing. Run ollama list to verify the model downloaded successfully.
Worth doing?
If you handle anything confidential, yes — it is the only setup where privacy is structural rather than contractual. If you use AI heavily and dislike the recurring bill, yes. If you want something that keeps working without an internet connection or a vendor's permission, yes.
If you mainly need the strongest possible reasoning on non-sensitive material, a cloud model is still the better tool, and there is no shame in that. Setup takes about twenty minutes either way, which is a cheap experiment for finding out which camp you are in.
Frequently asked questions
What are the minimum requirements to run AI locally?
8 GB of RAM is the practical floor, and it limits you to small models of roughly 3 to 4 billion parameters. 16 GB is the realistic recommendation and runs 7 to 8 billion parameter models comfortably, which is where most people find local AI becomes genuinely useful. 32 GB or more allows larger models. A dedicated GPU makes responses substantially faster but is not required — modern CPUs and Apple Silicon run smaller models at usable speed.
Is running AI locally actually free?
Yes. Ollama and the models it hosts are free to download and free to run, with no per-token cost and no subscription. The real costs are indirect: disk space, several gigabytes per model, and electricity, which is negligible for personal use. You also spend more of your own time on setup than you would signing up for a cloud service.
Is a local AI model as good as ChatGPT or Claude?
No, and it is worth being clear about that. A local 8 billion parameter model is noticeably weaker at complex reasoning, long multi-step tasks, and specialised knowledge than a frontier cloud model. What it is perfectly good at is summarising documents, drafting and rewriting text, extracting information, answering questions about your own files, and general conversation. For a large share of everyday use the gap is smaller than people expect.
Does local AI work without an internet connection?
Yes, once the model is downloaded. The initial download requires internet, but after that inference runs entirely on your own hardware. You can disconnect completely and the model continues to work — which is the simplest way to prove to yourself that nothing is being sent anywhere.
Can I run AI locally on Windows?
Yes. Ollama ships a native Windows installer and no longer requires WSL2. The OpenClaw Desktop App also runs natively on Windows. The setup is the same as on macOS: install the runner, pull a model, connect an interface.
Which local model should I use in 2026?
Match the model to your memory rather than chasing the newest release. On 8 GB, small models in the 3 to 4 billion parameter range from the Llama, Qwen or Gemma families are the sensible choice. On 16 GB, 7 to 8 billion parameter models are the sweet spot. On 32 GB or more you can run larger variants that get meaningfully closer to cloud quality. Model releases move quickly, so check what is current in Ollama's library rather than trusting any fixed recommendation.
Is local AI more private than a cloud service?
Substantially, yes. With a local model, your prompts and any file contents the model reads never leave your computer — there is no API call to inspect, log or retain. This is the difference that matters for client work, source code, medical or legal documents, and anything under a confidentiality obligation. It is also the only setup where privacy is a property of the architecture rather than a promise in a policy document.