# Gemini commerce benchmark: 4 models, 3 thinking budgets

> We compared Gemini 3.5, 3.7 and 3.8 Flash with 4 Argon at Low, Medium and High thinking on 36 tasks. Here's what the extra budget bought.

We use Gemini to bring a store associate over iMessage to every customer after their item arrives. The agent finds products, checks orders and handles returns, and it often calls several tools before writing a single text. We wanted to see what extra budget buys: a bigger model, or more thinking?

The launch of [Gemini 4 Argon](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/gemini/4-argon) gave us a reason to answer both at once. We ran 3.5 Flash, 3.7 Flash, 3.8 Flash and 4 Argon at Low, Medium and High thinking: 12 configurations on the same 36 tasks. The short version: Medium thinking matched or beat every other budget on passes in every model, and Argon bought the most passes (34/36) at 2.5x the price of 3.8 Medium.

## Results

### What each thinking budget buys

Pass rates and agent costs for completed tasks. Each model was tested at Low, Medium and High thinking on the same 36 tasks.

### All 4 models at all 3 thinking budgets

36 completed tasks per configuration. Lower cost and latency are better; the highlighted row has the highest pass rate in this sample.

| Model family | Thinking | Pass rate | Avg. judge | Agent cost/task | Median latency / task | p95 latency / task |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| 3.5 Flash | Low | 83.3% | 4.639 | $0.1312 | 65.49s | 295.97s |
| 3.5 Flash | Medium | 83.3% | 4.583 | $0.1659 | 55.14s | 184.33s |
| 3.5 Flash | High | 77.8% | 4.556 | $0.1808 | 129.02s | 271.93s |
| 3.7 Flash | Low | 80.6% | 4.694 | $0.1248 | 40.46s | 119.65s |
| 3.7 Flash | Medium | 88.9% | 4.750 | **$0.1157** | 44.49s | 139.38s |
| 3.7 Flash | High | 83.3% | 4.778 | $0.1496 | 43.56s | 146.66s |
| 3.8 Flash | Low | 83.3% | 4.694 | $0.1229 | **32.89s** | **95.54s** |
| 3.8 Flash | Medium | 88.9% | 4.944 | $0.1381 | 44.97s | 113.06s |
| 3.8 Flash | High | 83.3% | 4.639 | $0.1893 | 80.03s | 199.50s |
| 4 Argon | Low | 88.9% | 4.861 | $0.4218 | 36.97s | 281.47s |
| **4 Argon** | **Medium (Highest pass rate in this sample)** | **94.4%** | **5.000** | $0.3512 | 36.33s | 98.40s |
| 4 Argon | High | 83.3% | 4.750 | $0.3995 | 73.85s | 208.55s |

## What the extra budget buys

More thinking should cost more and, you'd hope, pass more tasks. High only did the first part: it cost more than Medium in every model and passed fewer tasks every time, and it was especially shaky on customer replies (Argon High passed 7/12 of those, against 10/12 at Medium). Low was the bigger surprise, because Argon Low and 3.7 Low both cost more than their Medium settings.

For Argon, the reason is tools. Thinking less made it flail: Argon Low averaged 13.36 tool calls per task against Medium's 8.25, and those extra round trips ate more than the thinking tokens saved (3.7 Low did a milder version, 5.92 calls vs 5.47). Low paid off best on 3.8, which became the fastest setup we tested (32.89-second median per task, 95.54-second p95) at 30/36 for $0.1229.

Across the Flash models, 3.5 Medium passed 30/36 for $0.1659 per task. Both 3.7 Medium and 3.8 Medium passed 32/36, at $0.1157 and $0.1381 respectively, making 3.7 Medium the cheapest of all 12 configurations.

### Suite results across thinking levels

Passed tasks in Low / Medium / High order, using the same tasks for every configuration.

| Selected suite | Tasks | 3.5 Flash L / M / H | 3.7 Flash L / M / H | 3.8 Flash L / M / H | 4 Argon L / M / H |
| --- | ---: | ---: | ---: | ---: | ---: |
| Catalog and shipping evidence | 3 | 2 / 2 / 2 | 2 / 2 / 2 | 1 / 3 / 3 | 3 / 3 / 3 |
| Support hours | 1 | 1 / 1 / 1 | 1 / 1 / 1 | 1 / 1 / 1 | 1 / 1 / 1 |
| Purchase assistance | 6 | 5 / 5 / 3 | 4 / 5 / 4 | 5 / 5 / 5 | 5 / 6 / 5 |
| Product measurements | 4 | 3 / 3 / 3 | 4 / 4 / 4 | 4 / 4 / 4 | 4 / 4 / 4 |
| Customer reply quality | 12 | 9 / 10 / 10 | 8 / 10 / 9 | 9 / 9 / 8 | 10 / 10 / 7 |
| Return policy and confirmation | 7 | 7 / 6 / 6 | 7 / 7 / 7 | 7 / 7 / 6 | 6 / 7 / 7 |
| Conversation ownership | 3 | 3 / 3 / 3 | 3 / 3 / 3 | 3 / 3 / 3 | 3 / 3 / 3 |

3.8 Medium's gains came mostly from catalog and shipping evidence (3/3, where 3.5 and 3.7 got 2/3) and, against 3.5, product measurements (4/4 vs 3/4 at every 3.5 setting), the kind of exact-detail question shoppers ask by text. Purchase assistance is where Argon Medium separates itself, as the only configuration to go 6/6. That's the real case for Argon: 34/36 overall for $0.3512 per task, with a 36.33-second median that sits between 3.8 Low and Medium, so it's plenty fast. The catch is price, since its 2 extra passes over 3.8 Medium cost 2.5x as much per task.

Even the winner wasn't perfect. Argon Medium failed 2 tasks that the judge scored a 5, because our pass rule also requires every tool and action check to pass. The [downloadable results](/blog/gemini-4-argon-main-agent-eval/results.json) have every configuration's detailed measurements if you want to dig in.

## How we tested

Each setting ran the same 36 shopping, returns and support tasks on October 13–14, 2026, with fixed agent code, prompts, tools and helpers. A pass required all tool/action checks and a score of 4 or 5 from the same 3.7 Flash High judge. Results summarize completed tasks, including agent and helper costs. This is a small sample with 1 completed run per task, and the judge may favor its own family.

## Takeaway

For an agent like ours, start at Medium: it tied or beat every other budget on passes in every model (3.5 Low tied it at 30), and High never beat it. If you want the fastest replies, use 3.8 Low; the cheapest, 3.7 Medium; the most passes, Argon Medium, if 2.5x the cost is worth 5-10% better pass-rate.