Follow-up: I Tested the Wrong Local Models. Gemma 4 Changes the Picture.
2026-11-21
My first post tested local models that were already a year and a half old. With a current model, Gemma 4 26B A4B, the same laptop gave answers that were mostly right instead of mostly wrong. Local AI is still not something I'd trust blindly, but the gap to the frontier models is much smaller than I reported.
As before, I ran the models and the tests myself. The setup, the choice of model and the analysis came from Claude, Anthropic's AI assistant. That includes the analysis of Claude's own output in the comparisons below, which may be biased in its favour.
What went wrong
The models I tested, Qwen3-30B-A3B and Qwen3-14B, were Claude's recommendation. It suggested them from memory, without checking what was current. Those models were released in spring 2025, and by November 2026 there were newer generations: Google's Gemma 4 (April 2026) and later Qwen releases.
When I asked again, Claude searched first and suggested Gemma 4 instead. So my first post compared 2025 local models with a 2026 frontier model, and the gap looked bigger than it is.
The lesson applies beyond AI: open models move fast, so check what's current before testing. A recommendation, even a confident one, can be out of date.
The new model and setup
Gemma 4 26B A4B is a mixture-of-experts model, like the Qwen3-30B-A3B, but it uses about 4 billion parameters per token instead of about 3 billion. At 4-bit quantization the file is 15.8 GiB. I used Unsloth's build with the same llama.cpp/Vulkan setup as before:
llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M \
-ngl 99 -c 16384 --host 127.0.0.1 --port 8080
The first attempt didn't fit. By default the kernel lets the integrated GPU use only half of the RAM: about 15 GiB of my 32 GB. llama.cpp's --n-cpu-moe 8 option kept part of the model on the CPU and made it run, but a model split between CPU and GPU writes more slowly.
The proper fix was to raise the GPU's memory limit to 24 GB with two kernel parameters, counted in 4 KB pages:
ttm.pages_limit=6291456 ttm.page_pool_size=6291456
I checked that the parameter exists on my kernel first (ls /sys/module/ttm/parameters/), added it to the GRUB command line and rebooted. After that, the whole model fits on the GPU, with about 8 GB left for the system and Firefox.
Speed: not faster, but fast enough
Gemma reads input faster than any model I tried, and writes at 14.6 tokens per second, which is faster than most people read. Same benchmark as before: llama-bench, 512-token prompt, 128 generated tokens, 3 runs, Firefox open.
| Model | Backend | Reading input (t/s) | Writing output (t/s) |
|---|---|---|---|
| Qwen3-30B-A3B | GPU, 40 of 48 layers | 181.1 | 18.5 |
| Qwen3-14B | GPU, all layers | 155.7 | 9.2 |
| Gemma 4 26B A4B | GPU, all layers | 224.3 | 14.6 |
It writes more slowly than the Qwen3-30B-A3B, as expected. On this laptop writing speed is limited by memory bandwidth, and Gemma reads about a third more weights for every token.
That's also why I didn't try the most accurate current local option, the dense Qwen3.8 27B. At an estimated 4 to 5 tokens per second on this hardware, it would be too slow to be useful to me.
Fitness test: right exercises, small slips
Gemma answered the question both Qwen models failed. I asked for vertical press exercises that are gentle on the shoulders, and it gave three that really are: the landmine press, the neutral-grip dumbbell press and the half-kneeling single-arm kettlebell press. The descriptions are correct, and the advice around them is standard coaching practice: press with the elbows about 30° forward (the "scapular plane"), don't arch the lower back, and balance pressing with pulling.
The old pattern is still there, just smaller:
- A setup that contradicts itself. For the dumbbell press it says to hold the dumbbells "at your sides with palms facing your ears". You can't do both. The press starts at shoulder height.
- Two exercises merged. What it calls "wall slides" (back against the wall, backs of the hands touching it) is usually called a wall angel. The muscle it says it trains, the serratus anterior, is the target of the wall slide done facing the wall.
- Simplifications stated as facts. "Most shoulder pain occurs with a pronated grip" is a generalization, and rules like "never" flare the elbows are rules of thumb.
| Qwen3-30B-A3B | Qwen3-14B | Gemma 4 26B A4B | |
|---|---|---|---|
| Suggests real vertical presses | No | No | Yes (3) |
| Exercises described correctly | No | Partly | Yes, with one setup slip |
| Mixes up exercises | Yes (floor press / chest-supported row) | Yes (floor press / push press) | Slightly (wall slide / wall angel) |
| Safe to follow | No | No | Mostly, if you already know the basics |
These are mistakes a reasonably experienced person would catch, and none would get anyone hurt. That wasn't true of the older models.
Backup script: correct logic, risky defaults
Gemma is the first local model whose script would actually do the backup. The task was the same: unlock an encrypted external disk, rsync the whole system to it, then close it. One difference from the first round: I didn't mention Alpine in my first prompt to Gemma, so its first version was written for bash on Ubuntu or Debian. I then asked for a POSIX shell version for Alpine, and Gemma produced a second one.
What it got right:
- The excludes work. The first version used the bash brace syntax that broke the Qwen3-14B's script, but written correctly, with commas. When I asked for POSIX shell, it rewrote them as separate
--exclude=options and explained whyshhas no brace expansion. - It keeps what a restore needs:
-aAXHpreserves ACLs, extended attributes and hard links. - It checks each step before the next, and cleans up if the mount fails.
- The POSIX version fixed exactly what I asked, without adding new bugs.
What it got wrong, in both versions:
--deleteonto the root of the disk. Anything else stored on that disk would be wiped on the first run. Claude's script synced into a subfolder for this reason.- It always reports success. Failed rsync, unmount or close still ends with "Process complete" and exit code 0.
- No cleanup on Ctrl+C. An interrupted run leaves the disk mounted and unlocked.
The POSIX revision says something new about the model: it corrects itself well when told what's wrong. But it only fixed what I pointed out. The --delete and cleanup problems were still there for me to find.
A reminder: Claude wrote this analysis, including of its own script in the last column.
| Qwen3-30B-A3B | Qwen3-14B | Gemma 4 (POSIX) | Claude | |
|---|---|---|---|---|
| Full system backup that works | No | No | Yes | Yes |
| Preserves permissions fully | No | Mostly | Yes (no --numeric-ids) | Yes |
| Safe for other data on the disk | n/a | n/a | No | Yes (subfolder) |
| Cleans up on Ctrl+C | No | No | No | Yes |
| Honest "done" message | Yes | Yes | No | Mostly |
| Usable as written | No | No | After fixes, with care | After small fixes |
What changes in my conclusion
My first post ended stuck: local models on my hardware weren't good enough, and frontier models don't give me privacy assurances. With a current model, the first half of that is no longer true in the same way.
What changes:
- Local is now a real option for part of the work. Gemma's mistakes are small enough that, for tasks where I review the result anyway, a local model is useful rather than a liability. That covers several of the uses I listed in the first post: organizing my archive, structuring unstructured documents and summarizing my own notes. All of these keep the data on my laptop.
- The speed is acceptable. At 14.6 tokens per second it's comfortable to use, on a laptop with no discrete GPU.
What doesn't change:
- I still can't trust what I can't check. The errors are smaller, but they're the same kind: confident, well formatted, and visible only if I already know the answer. Advice on health or finances, where I can't easily verify the details, still needs a better model or a human.
- Even the frontier model needed a line-by-line review. Claude's backup script had its own hidden flaw. Reviewing everything applies to every model.
So I'm less stuck than I was. Local AI, used for grounded tasks on my own data and with a review of everything it produces, now fits my principles and is good enough to be worth using. For the rest, the dilemma remains. And next time, I'll check what's current before I test.