Skip to content
Damien K. Edwards

August 9, 2026 · 2 min read

Running Gemma 4 Locally — What It's Actually Like

local-llmllama-cppai-engineeringopen-source

Google's Gemma 4 is one of the most capable open-weight models available for local inference right now. Unlike its predecessors, Gemma 4 ships in a multimodal form factor — it can handle text and images — and the smaller variants are genuinely usable on consumer hardware.

Here's what running it locally actually looks like.

What You Need

The sweet spot for local Gemma 4 is the 4B or 12B parameter variant, quantized to Q4 or Q8:

  • 4B Q8 — ~4.5GB RAM. Runs on most machines with 8GB+
  • 12B Q4 — ~7GB RAM. Needs 16GB+ for comfortable use
  • 27B Q4 — ~16GB RAM. GPU recommended

For CPU-only inference, the 4B hits a usable speed. The 12B is noticeably slower but worth it if you have the RAM.

Setting It Up with llama.cpp

The fastest path to running Gemma 4 locally is llama.cpp:

llama-server --hf-repo bartowski/gemma-3-4b-it-GGUF:Q8_0 -c 8192

Once running, it exposes an OpenAI-compatible endpoint at http://localhost:8080/v1 — drop it into any tool that accepts a custom base URL.

What It's Good At

Gemma 4 punches above its weight on:

  • Instruction following — clean, direct responses without padding
  • Code generation — solid for Python, JavaScript, and SQL
  • Summarization — handles long documents well at the 12B scale
  • Multimodal reasoning (IT variant) — describe images, answer questions about screenshots

Where It Falls Short

It's not Claude. Complex multi-step reasoning, nuanced writing, and anything requiring deep world knowledge will remind you of the gap. For a local model though, it's the closest thing to a daily driver I've used.

The Real Use Case

The argument for running Gemma 4 locally isn't raw capability — it's privacy, cost, and availability. Sensitive documents that shouldn't leave your network, high-volume tasks where API costs add up, or just having something available when you're offline.

Pair it with a router like LiteLLM or Hermes and you get seamless fallback between local and cloud models — local for the routine stuff, cloud for the hard problems.

Related