Mário Nogueira

Follow-up: I Tested the Wrong Local Models. Gemma 4 Changes the Picture.

2026-11-21

My first post tested local models that were already a year and a half old. With a current model, Gemma 4 26B A4B, the same laptop gave answers that were mostly right instead of mostly wrong. Local AI is still not something I'd trust blindly, but the gap to the frontier models is much smaller than I reported.

As before, I ran the models and the tests myself. The setup, the choice of model and the analysis came from Claude, Anthropic's AI assistant. That includes the analysis of Claude's own output in the comparisons below, which may be biased in its favour.

What went wrong

The models I tested, Qwen3-30B-A3B and Qwen3-14B, were Claude's recommendation. It suggested them from memory, without checking what was current. Those models were released in spring 2025, and by November 2026 there were newer generations: Google's Gemma 4 (April 2026) and later Qwen releases.

When I asked again, Claude searched first and suggested Gemma 4 instead. So my first post compared 2025 local models with a 2026 frontier model, and the gap looked bigger than it is.

The lesson applies beyond AI: open models move fast, so check what's current before testing. A recommendation, even a confident one, can be out of date.

The new model and setup

Gemma 4 26B A4B is a mixture-of-experts model, like the Qwen3-30B-A3B, but it uses about 4 billion parameters per token instead of about 3 billion. At 4-bit quantization the file is 15.8 GiB. I used Unsloth's build with the same llama.cpp/Vulkan setup as before:

llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M \
  -ngl 99 -c 16384 --host 127.0.0.1 --port 8080

The first attempt didn't fit. By default the kernel lets the integrated GPU use only half of the RAM: about 15 GiB of my 32 GB. llama.cpp's --n-cpu-moe 8 option kept part of the model on the CPU and made it run, but a model split between CPU and GPU writes more slowly.

The proper fix was to raise the GPU's memory limit to 24 GB with two kernel parameters, counted in 4 KB pages:

ttm.pages_limit=6291456 ttm.page_pool_size=6291456

I checked that the parameter exists on my kernel first (ls /sys/module/ttm/parameters/), added it to the GRUB command line and rebooted. After that, the whole model fits on the GPU, with about 8 GB left for the system and Firefox.

Speed: not faster, but fast enough

Gemma reads input faster than any model I tried, and writes at 14.6 tokens per second, which is faster than most people read. Same benchmark as before: llama-bench, 512-token prompt, 128 generated tokens, 3 runs, Firefox open.

ModelBackendReading input (t/s)Writing output (t/s)
Qwen3-30B-A3BGPU, 40 of 48 layers181.118.5
Qwen3-14BGPU, all layers155.79.2
Gemma 4 26B A4BGPU, all layers224.314.6

It writes more slowly than the Qwen3-30B-A3B, as expected. On this laptop writing speed is limited by memory bandwidth, and Gemma reads about a third more weights for every token.

That's also why I didn't try the most accurate current local option, the dense Qwen3.8 27B. At an estimated 4 to 5 tokens per second on this hardware, it would be too slow to be useful to me.

Fitness test: right exercises, small slips

Gemma answered the question both Qwen models failed. I asked for vertical press exercises that are gentle on the shoulders, and it gave three that really are: the landmine press, the neutral-grip dumbbell press and the half-kneeling single-arm kettlebell press. The descriptions are correct, and the advice around them is standard coaching practice: press with the elbows about 30° forward (the "scapular plane"), don't arch the lower back, and balance pressing with pulling.

The old pattern is still there, just smaller:

Qwen3-30B-A3BQwen3-14BGemma 4 26B A4B
Suggests real vertical pressesNoNoYes (3)
Exercises described correctlyNoPartlyYes, with one setup slip
Mixes up exercisesYes (floor press / chest-supported row)Yes (floor press / push press)Slightly (wall slide / wall angel)
Safe to followNoNoMostly, if you already know the basics

These are mistakes a reasonably experienced person would catch, and none would get anyone hurt. That wasn't true of the older models.

Backup script: correct logic, risky defaults

Gemma is the first local model whose script would actually do the backup. The task was the same: unlock an encrypted external disk, rsync the whole system to it, then close it. One difference from the first round: I didn't mention Alpine in my first prompt to Gemma, so its first version was written for bash on Ubuntu or Debian. I then asked for a POSIX shell version for Alpine, and Gemma produced a second one.

What it got right:

What it got wrong, in both versions:

The POSIX revision says something new about the model: it corrects itself well when told what's wrong. But it only fixed what I pointed out. The --delete and cleanup problems were still there for me to find.

A reminder: Claude wrote this analysis, including of its own script in the last column.

Qwen3-30B-A3BQwen3-14BGemma 4 (POSIX)Claude
Full system backup that worksNoNoYesYes
Preserves permissions fullyNoMostlyYes (no --numeric-ids)Yes
Safe for other data on the diskn/an/aNoYes (subfolder)
Cleans up on Ctrl+CNoNoNoYes
Honest "done" messageYesYesNoMostly
Usable as writtenNoNoAfter fixes, with careAfter small fixes

What changes in my conclusion

My first post ended stuck: local models on my hardware weren't good enough, and frontier models don't give me privacy assurances. With a current model, the first half of that is no longer true in the same way.

What changes:

What doesn't change:

So I'm less stuck than I was. Local AI, used for grounded tasks on my own data and with a review of everything it produces, now fits my principles and is good enough to be worth using. For the rest, the dilemma remains. And next time, I'll check what's current before I test.

Sources