Defining Local AI

• A fully local setup requires both the model file and the runtime tool to reside on your hardware, ensuring no data reaches an external server.2:10
• You can verify locality by disabling Wi-Fi; if the AI functionality continues to operate, the system is genuinely local.
• Many popular apps—such as Cursor or Claude Code—are local tools that interact with cloud models, meaning they do not provide the privacy benefits of an offline-only model.1:36

The Performance Mechanism

• Your system's fast memory capacity—VRAM for discrete GPUs or unified memory for modern Macs—determines the size of the model you can load.8:28
• Memory bandwidth is the primary constraint on inference speed, measured in tokens per second; high capacity alone does not guarantee performance.9:22
• Quantization, such as 4-bit (Q4) compression, allows models to run on consumer hardware by shrinking them to roughly one-quarter of their original size with minimal quality degradation.5:04

Hardware Recommendations

• For most users, frontier-scale models are impractical; targeting ~8B–35B parameter models provides the most efficient balance of utility and speed.8:07
• A budget entry point is a used RTX 3060 12GB or base Mac mini, while a mid-tier system—like an RTX 3090 24GB or M4 Pro 48GB Mac mini—is the threshold where local AI transitions from a toy to a useful tool for coding and agents.11:12
• High-end setups, such as 128GB unified-memory systems, are necessary to run 120B-class models, though inference speed remains sensitive to the specific bandwidth limitations of the chip.13:32