Running Models Locally¶
Every assistant in the tools directory runs in a vendor's cloud: what you type travels to their servers. There is another way. The open-weights models at the bottom of the directory can be downloaded and run entirely on your own computer, where nothing you type leaves the machine. This page is a practical starting point.
Why bother. Three reasons people run models locally: privacy (your text never leaves your hardware), cost (the models and the tools below are free), and learning (nothing demystifies a language model like running one yourself). The honest trade-off: local models are smaller and noticeably less capable than frontier cloud models, and they run slower. For drafting an email that is fine; for hard reasoning you will notice the gap.
Running a model locally takes the vendor out of the picture: nothing you type is collected, stored, or used for training by a third party, which is the risk the AI Responsible Use Policy describes for public tools. Privacy law follows the data rather than the tool, so patient information and student records on your own machine are still covered by the Health Insurance Portability and Accountability Act (HIPAA) and the Family Educational Rights and Privacy Act (FERPA), as the policy notes.
Where your text goes
Cloud assistant
your text
the vendor's servers
Local model
your text stays on this machine
Nothing leaves; works offline.
Rented GPU
your text
a server you rent
What You Need¶
Memory is the main constraint. A model has to fit in your computer's memory (RAM), or better, in the video memory of a graphics card (a graphics processing unit, or GPU). Most local models are used in quantized form, meaning compressed versions that trade a small amount of quality for a much smaller size. Rough expectations:
- A typical laptop (8 GB of memory) runs small models, around 4 billion parameters and under. Fine for experimenting, summaries, and simple drafting.
- 16 GB of memory runs mid-size models. OpenAI's gpt-oss-20b is designed to fit in 16 GB, though on a laptop where the operating system and open apps share that memory it fits only with little else open.
- 32 GB of memory, or a graphics card with 16 to 24 GB of video memory, opens up the 20 to 30 billion parameter class, where local models start feeling genuinely useful. Mixture-of-experts models such as gpt-oss-20b fit in less; Hardware for Local AI explains why.
- Apple silicon Macs share memory between the processor and graphics, which makes them popular for local models; the same memory math applies.
For the full picture (what tokens per second feels like, why video memory beats system memory, quantization trade-offs, mixture-of-experts models, and an interactive estimator for your own machine), see Hardware for Local AI.
The Simple Path: A Local Chat Assistant¶
- Install a runner. LM Studio is the simplest start: a desktop application where you browse models, click download, and chat, no command line involved. Ollama works from the command line and also has its own desktop chat app, and it pairs with Open WebUI if you want a browser chat interface on top. All three are free for local use and listed in the directory.
- Pick a small model first. Start with something in the 4 billion parameter class (a small Gemma or Qwen variant), confirm it runs smoothly, then work upward to the largest model your memory allows. The open-weights section lists the major families, the Benchmarks page tracks how they currently rank, and both runners show curated, ready-to-download versions of all of them.
- Calibrate expectations. Replies stream more slowly than cloud assistants, knowledge cutoffs are real, and there is no web search unless you add one. Smaller local models make the same kinds of mistakes as cloud assistants, and more of them; the misconceptions page covers what to watch for.
Beyond Chat: Images, Video, and Voice¶
Text is the simple case. Open-weights models also exist for image generation, video generation, speech-to-text, and text-to-speech, and a different tool is standard there: ComfyUI, a free, open-source application where you assemble model pipelines visually by connecting nodes on a canvas. It is the standard way to run the Stable Diffusion family and newer open image and video models locally. The learning curve is steeper than a chat runner, and image and video models generally want more video memory than text models, but the community templates make the first steps manageable.
For voice, Whisper (already in the directory) transcribes speech to text entirely on your machine, and open text-to-speech models are improving quickly.
When Your Machine Cannot Keep Up: Renting a GPU¶
If a model you want will not fit in your hardware, you can rent the hardware instead: cloud providers such as Amazon Web Services (AWS) offer GPU instances by the hour, and GPU rental marketplaces like RunPod and Vast.ai make the same thing simpler and usually cheaper, with one-click templates for Ollama, Open WebUI, and ComfyUI. You get capability no laptop can match and pay only while the machine runs.
The moment your model runs on rented hardware, your data leaves your machine, which was the headline reason to run locally in the first place. A rented GPU running an open-weights model is still more under your control than a consumer chatbot (you choose the model, nothing is retained to train on by default, and you can destroy the instance), but the data now sits in a provider's data center, so the AI Responsible Use Policy's section on sensitive information applies as it would to any hosted service. Add the practical frictions (per-hour billing that keeps running if you forget to shut down, and more setup than a desktop app) and the honest summary is: rent for capability and experiments, run truly locally for privacy, and use the directory's cloud tools for everyday work.
Watch: Setting It Up¶
Verified walkthroughs for two of the tools on this page. Links reviewed September 2026; check a video's date against the tool's current version.
Beyond these, several channels followed by our Videos page regularly test new open-weights releases on consumer hardware, which is the fastest way to see what a model of a given size can actually do before downloading anything. For agents rather than models, see the companion guide on AI Agents.
Content last reviewed September 1, 2026. Review dates are maintained as described on the About page.