Local AI Guide

Run state-of-the-art AI on your own hardware.

Full privacy, zero per-token fees. From a fresh laptop to a running LLM in about 15 minutes.

Why run AI locally?

Privacy

Your prompts never leave your machine.

Zero cost

No API charges. No rate limits.

Offline

Works on a plane. Works without internet.

Full control

Pick the model, weights, and system prompt.

Hardware you need

Model size	Minimum RAM	Recommended GPU	Speed
3B – 8B	8 GB	None / integrated	20-50 tok/s CPU
13B – 27B	16 GB	8 GB VRAM (RTX 3060)	15-30 tok/s
70B (quantized)	32-48 GB	24 GB VRAM (RTX 4090 / M3 Max)	10-20 tok/s
405B	64+ GB unified	M3 Ultra or 2x A100	5-10 tok/s

Apple Silicon punches above its weight thanks to unified memory. A 64 GB M3 Max runs Llama 3.3 70B faster than most expensive gaming GPUs.

Quantization and GGUF

A raw 70B model is roughly 140 GB. Quantization shrinks it by storing weights at lower precision (4-bit, 5-bit, 8-bit) with almost no quality loss for most tasks. The .gguf is the community standard for quantized models.

A good default: Q4_K_M. It cuts size by ~4x while preserving 98%+ of the quality.