Can I run Phi-3.5-mini?

3.8B with 128K context. No GQA, so KV grows fast at long context.

generalchatreasoning · sizes: 3.8B · MIT

What runs it

machinebest fitverdictspeed
Apple M1 · 8GB 3.8B Q5_K_M 🐢 CPU-only (slow) 2–3 tok/s
Apple M1 Pro · 16GB 3.8B Q8_0 ✅ Runs (tight) 25–40 tok/s
Apple M2 · 16GB 3.8B Q8_0 ✅ Runs (tight) 14–25 tok/s
Apple M3 Pro · 18GB 3.8B Q8_0 ✅ Runs (tight) 20–35 tok/s
Apple M2 Pro · 32GB 3.8B Q8_0 ✅ Runs comfortably 25–45 tok/s
Apple M4 Pro · 48GB 3.8B Q8_0 ✅ Runs comfortably 35–55 tok/s
Apple M3 Max · 36GB 3.8B Q8_0 ✅ Runs comfortably 45–75 tok/s
Apple M2 Max · 64GB 3.8B Q8_0 ✅ Runs comfortably 45–75 tok/s
Apple M4 Max · 128GB 3.8B Q8_0 ✅ Runs comfortably 55–95 tok/s
Apple M2 Ultra · 192GB 3.8B Q8_0 ✅ Runs comfortably 75–130 tok/s
NVIDIA GeForce RTX 3060 · 12GB VRAM 3.8B Q8_0 ✅ Runs comfortably 35–60 tok/s
NVIDIA GeForce RTX 4070 · 12GB VRAM 3.8B Q8_0 ✅ Runs comfortably 50–85 tok/s
NVIDIA GeForce RTX 4090 · 24GB VRAM 3.8B Q8_0 ✅ Runs comfortably 100–170 tok/s
NVIDIA GeForce RTX 5080 · 16GB VRAM 3.8B Q8_0 ✅ Runs comfortably 100–160 tok/s
NVIDIA GeForce RTX 5090 · 32GB VRAM 3.8B Q8_0 ✅ Runs comfortably 180–300 tok/s
AMD Radeon RX 7900 XTX · 24GB VRAM 3.8B Q8_0 ✅ Runs comfortably 95–150 tok/s

Shopping for a machine? What hardware do I need to run Phi-3.5-mini? →

Check your exact rig: open the checker →

$npx runlocal-sh can-i-run phi-3.5-mini