How good is Gemini 3.8 Flash at commerce?
Gemini 3.8 Flash Medium tied our previous winner of 3.7 High, but higher thinking on 3.8 was not all that good
Ilya Valmianski 
Models have been coming out at breakneck speed and we’re always excited to see how well they do for our commerce agent. We’ve already benchmarked Gemini 3.5, 3.6, and 3.7 Flash on our main agent and Gemini 3.5 Flash-Lite on our catalog agent. Two days ago (a century ago in AI news time), Google launched Gemini 3.8 Flash, so we asked the same question we ask every time: does a newer model actually make our customer agent better?
The short answer is “kind of?” Because our commerce agent is a bit off-target from what models are usually good at, a version bump isn’t always an upgrade for us. We skipped 3.6 entirely because it underperformed 3.5. 3.8 is an improvement, but only in the sense that 3.8 Medium matched the quality of 3.7 High while returning main-model calls 27% faster. 3.8 High didn’t add anything durable on top of that: it picked up one extra pass in the first run, and the lead evaporated when we reran the cases that split them.
Results
Every configuration ran the same 35 production-shaped tasks (methods are in the 3.7 post, with a short recap below). We changed only the primary model and its pinned thinking level. Catalog, helper models, simulator, judge, prompts, tools, and safety rules stayed fixed within a run.
Gemini Flash customer-agent benchmark, 3.5 through 3.8
35 tasks per configuration. Latency is the median individual call to the primary model, excluding Shopify, tools, helpers, simulator, and judge; the August rows were recomputed onto that basis from preserved request timings. *3.7 High was rerun in September alongside 3.8 as a fresh control on the current agent prompt. Lower cost and latency are better.
| Model family | Thinking | Pass rate | Avg. judge | Standard cost / run | Median main-model call |
|---|---|---|---|---|---|
| 3.5 Flash | Low | 65.7% | 4.143 | $0.2012 | 5.66s |
| Medium | 74.3% | 4.371 | $0.2534 | 7.66s | |
| High | 74.3% | 4.400 | $0.2534 | 10.11s | |
| 3.6 Flash | Low | 71.4% | 4.286 | $0.0962 | 2.91s |
| Medium | 68.6% | 4.229 | $0.1164 | 5.40s | |
| High | 60.0% | 3.829 | $0.0874 | 5.69s | |
| 3.7 Flash | Low | 68.6% | 4.171 | $0.1125 | 3.01s |
| Medium | 68.6% | 4.143 | $0.1188 | 3.86s | |
| High | 74.3% | 4.400 | $0.1216 | 5.20s | |
| High (Sept rerun)* | 77.1% | 4.514 | $0.1219 | 7.88s | |
| 3.8 Flash | Low | 65.7% | 4.171 | $0.1202 | 2.36s |
| Medium Overall winner | 77.1% | 4.457 | $0.1280 | 5.75s | |
| High | 80.0% | 4.400 | $0.1493 | 7.48s |
3.8 High passed 28 tasks, while 3.8 Medium and the 3.7 High rerun passed 27. One task is too thin to call a winner on, especially with the judge scores pointing the other way (4.457 for Medium, 4.400 for High). So we reran every scenario where Medium and High differed in pass outcome or judge score, 3 times each. Exact tie, 8 of 15 to 8 of 15. The one task High won in the broad run passed 3 out of 3 for both settings in the rematch.
That leaves efficiency as the real difference. Medium used 42% fewer reasoning tokens per primary request than High, cost 14% less per run, and returned a main-model call 23% faster. Note that for 3.7 High we report both the results from our previous run and from today’s comparison. 3.7 has slowed down a bit; it’s unclear whether the model is genuinely slower or changes in our harness caused this. Importantly, the 3.7 High pass rate barely moved between the two runs (26 of 35 in August, 27 today).
What the agent had to do
The 35 tasks come from prior real-customer failures: 11 single-turn cases (returns, exchanges, product questions, mixed return-and-browse, review suppression), 12 multi-turn cases (context retention, return execution, follow-on purchases, price deltas, existing labels, review requests), and 12 quality regressions (delivery status, escalation timing, feedback handling, soft fit complaints, settled conversations). Multi-turn tasks use an LLM simulator as the customer. A task passes only when every deterministic check on tool use, grounding, and escalation passes and a separate judge scores the conversation 4 or 5 out of 5.
Costs include the primary loop and fixed helpers, exclude the simulator and judge, and keep every task in the denominator whether it passed or failed. We used Google’s public Standard pricing as of September 2, 2026: $0.75 per million input tokens, $0.075 per million cached-input tokens, and $3.75 per million output or reasoning tokens for 3.8 Flash. Those are introductory rates through December 31, 2026, so the economics shift when the published rates double on January 1, 2027.
Takeaway
Gemini 3.8 Flash is fine for commerce tasks but not a huge improvement over 3.7. The main change is that you can get away with fewer reasoning tokens for a similar level of performance.
