Qwen3.6-27B vs Gemma4-31B vs Qwen3.5-122B

  Qwen3.5-122B-Q4_K_M:
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/Qwen3.5-122B-A10B-Q4_K_M-00001-of-00003.gguf
      --mmproj /models/qwen3.5-122b/mmproj-BF16.gguf
      --ctx-size 64000
      --flash-attn on
      --cache-type-k q8_0
      --cache-type-v q8_0
      --ubatch-size 512
      --split-mode layer
      --parallel 1
      --spec-type ngram-mod

  gemma4-31b-Q8:
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/gemma-4-31B-it-Q8_0.gguf
      --model-draft /models/mtp-gemma-4-31B-it.gguf
      --mmproj /models/mmproj-BF16.gguf
      --n-gpu-layers 99
      --ctx-size 64000
      --flash-attn on
      --spec-type draft-mtp,ngram-mod
      --spec-draft-n-max 2
      --ubatch-size 512
      --split-mode tensor
      --fit off
      --parallel 1

  Qwen3.6-27B-Q8:
    cmd: >
      /usr/local/bin/llama-server
      --port ${PORT}
      --model /models/Qwen3.6-27B-Q8_0.gguf
      --mmproj /models/mmproj-Q8_0.gguf
      --n-gpu-layers 99
      --ctx-size 64000
      --flash-attn on
      --ubatch-size 512
      --spec-type draft-mtp,ngram-mod
      --spec-draft-n-max 2
      --split-mode tensor
      --fit off
      --parallel 1

Intro

Caveat before we begin: I’m well aware running 122b at a 4bit quant with Q8 kv isn’t giving it the best chance, but considering Q8_0 comes in at a cool ~130gb I genuinely don’t think its worth considering in this discussion. My biggest focus is what I can run at home, and while I technically have 144gb ram combined I’m not interested in waiting many hours as a response crawls in the single digit decode (not even to mention the prefill freeze). I’d also imagine anyone with the requisite VRAM would be running Deepseek V4 Flash instead at this point. I also will not be doing the full count for each benchmark, so these results won’t be comparable with the leaderboards. I simply don’t want to run tests for a week. I’ll mention how many tests were run for each benchmark.

Traditional Benchmarks

BFCL v4

Details: 50 samples, 6 test subsets → 300 samples total

My first priority is tool calling. My biggest complaint with Gemma4-31b is its apparent lack of desire to chain tool calls in my local use-cases. The Berkeley Function-Calling Leaderboard (BFCL) tool calling benchmark is the perfect place to get some understanding. The results showed only small deviation — not enough to exceed the confidence interval. These three models appear to be just about equal in this space. The gaps between 31B and 27B in Irrelevance and Live Multiple tests directly contradict my personal experiences. I’ll get more into why in my personal benchmarks.

IFEval

Details: 250 samples

The results of the Instruction-Following Evaluation were pretty middling with a marginally significant victory squeezed out by Gemma4-31B over the Qwens, which tied. One super interesting detail that does match my experience: Gemma4 uses notably fewer tokens for the same task. It averaged 21% fewer tokens on average (283 vs ~360).

MMLU-Pro

Details: 30 samples, 14 categories → 420 samples total

MMLU’s overall scores demonstrated a tie. Gemma4-31B did have a small accuracy lead, but definitely not significant at this test count. The latency differences were pretty notable and make sense. On my hardware, Gemma4-31B prefill and decode are a bit slower than Qwen3.6-27B, but it’s so much more succinct with its responses/thinking that it makes up for it. There were some back and forths in individual categories.

Arena-Hard

Details: 75 samples, run with --repeat-penalty 1.1 --min-p 0.05, judge by Deepseek V4 Pro

Gemma4-31B squeezed an advantage with a marginally higher win rate and the least average output tokens. 122B did the worst in terms of win rate. Even considering Gemma’s conciseness may be helping it with the judge and 122b only activates 10b, it being matched by models 1/4 its param count shows how substantial a couple months is in the LLM world right now. There were three super notable tests where Gemma aced it and the Qwens simply failed: JS deep object search, Python zip archiving, and yfinance data + plot.

HumanEval+ & MBPP+

Details: One pass each

• HumanEval+: 164 problems
• MBPP+: 378 problems

MBPP+ failed to differentiate the three models. All three are likely nearly saturated on that difficulty. On the other hand, Qwen3.6-27B really did not hold its own against the more expanded edge-case tests in HumanEval+. Qwen3.5-122B did quite well, and Gemma4-31B held a smaller but noteworthy 3.1% gap above. I was quite surprised at how poorly 27B did on this test in comparison. Most of the time, this is the exact context in which people claim it’s superior.

Word500

My personal benchmark. For those who aren’t familiar, see my more in-depth blog post covering this test: The Word500 Benchmark. It’s a really good test of a model’s reasoning capabilities and efficiency. I typically disregard speed on this benchmark. Going into this I knew from prior experience that Gemma4-31B would crush it, but I actually hadn’t run it on Q8 before. All three models passed each test, but as expected Gemma crushed the Qwens in token efficiency. 122B used fewer tokens on average than 27B.

Unstructured / Personal Tests

Thus far, the benchmarks have suggested that Gemma4-31B is simply the best choice between the three models. I’m really surprised because when I have tried it in my personal use-cases that hasn’t been the case. This next section will focus on such use-cases.

Open-WebUI Research Model

For chat, I primarily use an Open-WebUI Docker container. I have a handful of models defined there, but the most important function of both my General and Deep Research models is the ability to search the web. I provide that tooling through a SearXNG/Firecrawl stack, and most specifically I expect the model to:

• Decide when it needs to search_web, either for missing or up-to-date knowledge.
• Decide N, how many results it should get based on query complexity.
• Look through snippets, pick the best one, and use fetch_url to get the full context.
• Loop this as many times needed to compile an informed response

I also like the Task function Open-WebUI provides. It lets the model create_tasks and update_task to track its own progress and display to the user the stage its on. This is not a crucial component, but a really nice to have feature.

Will any of these models outperform the others in this context?

Setup

For this section, I’ll define three models in Open-WebUI - one for each of Gemma4-31B, Qwen3.6-27B, and Qwen3.5-122B. Each model will be provided the required tools, with Function Calling set to Native, and the following system prompt:

- Current date: {{CURRENT_DATE}}

## Planning & Task Tracking
- Before acting, think through a concise plan of steps.
- Use "create_tasks" (plural) to publish that plan as a checklist immediately.
- Use "update_task" (singular) after completing each step - including the final one. Never skip the last "update_task" call.

## Web Search:
- use when the answer depends on current events, recent releases, prices,
  versions, or anything that may have changed since your cutoff - or when you're
  unsure of a fact. Don't search for stable knowledge you already hold. Ground your
  answer in the returned results and cite sources. If results are empty or weak, say
  so instead of guessing.

  - When using search_web, set the count parameter (1–3) proportional to query complexity:
  • 1: Simple facts, definitions, single-entity lookups
  • 2-3: More complex/niche topics or highly specific technical queries
  - After search_web ALWAYS select the most promising result and call fetch_url to extract deeper context

The prompts used were:

Prompt 1: Search for a highly rated chocolate chip cookie recipe. Briefly summarize the author’s included anecdotal story and tell me exactly how many grams of butter it uses, and what temperature it says to set the oven to.

Prompt 2: Look up the 3 most popular Intel Core Ultra CPUs. Find the Passmark benchmark score and MSRP for each. Calculate the Price:Passmark ratio and the TDP:Passmark power efficiency ratios.

Prompt 3: Find out who currently holds the outdoor world record in the men’s 1 mile run, and identify the exact meet, venue, and date where it was set. Then, from a detailed report on that specific race, tell me: (a) the names of the pacemakers and roughly where in the race they dropped out, (b) the record-holder’s split times at 800m and 1500m, and (c) exactly how many seconds the new mark beat the previous world record by.

Each was judged on the following criteria:

1. Correctness - It should get accurate, real answers before synthesizing a response.
2. Efficiency - It should set N to a logical value based on complexity and make no more search or fetch calls than necessary.
3. Task generation - Did it create a plan and generate tasks?

Results

Gemma4-31B Q8_0

* The Intel CPUs prompt took two tries to reach success. The first attempt blasted through 17 searches, then simply ended the response with a thought block containing “The”.

Qwen3.6-27B Q8_0

* Like Gemma4, the Intel CPU prompt failed the first run. Qwen did many searches then ended with a malformed tool call on the first try.

Qwen3.5-122B Q4_K_M

* I guess my mile run prompt wasn’t as snippet-resistant as I thought; somehow 122B got the answers without a fetch.

These search query tests did not prove as useful or conclusive as I thought they would. In the past, when I’ve tried Gemma4-31B in this role, it was entirely reluctant to chain multiple tool calls of different kinds. It would search_web over and over and never follow up with a fetch. It would often simply tell me the information wasn’t provided in the snippets, and try to force a conclusion from its own knowledge. Some of the quirks still remain, like how it refuses to supply an N for the search calls, or close open tasks it created, but overall it is plenty successful at the loop. I assume some of the changes must be attributed to Google’s template improvements, though I had kept up with those in the past. The 122B Qwen3.5 was very efficient and controlled with the tool calls it made, even though it refused to interact with tasks just like Qwen3.6-27B did. I really like how 122B feels here, speed excluded. The lack of adherence to the search_web - N parameter and the sporadic spikes of a dozen-plus search calls do make Gemma4-31B a tough option for this, because it ends up sprinting toward a rate limit.

Pelican on a Bicycle

Let’s end on the classic silly “pelican on a bicycle” prompt to compare, why not.

Generate an SVG of a pelican riding a bicycle

SVG outputs from all three models: Qwen3.6-27B (top), Qwen3.5-122B (middle), Gemma4-31B (bottom)

I think these specific results speak for themselves - a pretty obvious gap on a likely “benchmaxxed” subjective test. 27B on top, followed by 122B, and Gemma4 dead last in a dramatic way.