# How good is Gemini 3.7 Flash at commerce?

> Gemini 3.7 Flash High matched our production Gemini 3.5 Flash configuration on 35 customer-agent tasks while cutting Standard-tier cost per run by 52%.

We use Gemini models to bring a store associate over iMessage to every customer after they receive their item. When [Gemini 3.7 Flash launched](https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-7-flash), we wanted to know if we should switch. Newer doesn't always mean better in the context of tasks that are not being hill-climbed by the labs, and for example [last month Gemini 3.5 Flash-Lite lost to 3.1 Flash-Lite](/blog/gemini-3-5-flash-lite-agent-eval) on our catalog agent. If we just upgraded because "it is newer" we would have gotten worse and more expensive. Our earlier tests on 3.6 Flash were also disappointing, which is why we stayed on 3.5 Flash for our main agent.

For this blog we'll show our bake-off between Gemini 3.5, 3.6, and 3.7 Flash, each at Low, Medium, and High thinking. 9 configurations, 315 customer-agent executions (full [task description and methods](#task-description-and-methods) at the end of the post). Thinking more is not always better, sometimes performance peaks at less than high (our current agent runs 3.5-medium because 3.5-high didn't outperform), sometimes higher thinking leads to overthinking and actually degrades performance. What was also interesting is that higher thinking was also not always more expensive, this is because higher thinking often meant better use of tools, and fewer tool turns often compensated more tokens per turn.

In our current test three configurations tied at the top with successfully completing 26 of 35 tasks: 3.5 Medium, 3.5 High, and 3.7 High. Among these, 3.7 High tied for the top judge score, and both cost about half as much and had half the latency per run.

## Results

Each configuration ran the same 35 production-shaped tasks against our main customer-facing agent, with everything except the primary model and its thinking level held fixed.

### Primary customer-agent thinking matrix

Matched results across 35 tasks per configuration. Lower cost and latency are better.

| Model family | Thinking | Pass rate | Avg. judge | Standard cost / run | Median latency / run |
| --- | --- | ---: | ---: | ---: | ---: |
| 3.5 Flash | Low | 65.7% | 4.143 | $0.2012 | 42.40s |
| 3.5 Flash | Medium | 74.3% | 4.371 | $0.2534 | 52.40s |
| 3.5 Flash | High | 74.3% | 4.400 | $0.2534 | 55.53s |
| 3.6 Flash | Low | 71.4% | 4.286 | $0.0962 | 28.64s |
| 3.6 Flash | Medium | 68.6% | 4.229 | $0.1164 | 33.42s |
| 3.6 Flash | High | 60.0% | 3.829 | $0.0874 | 30.82s |
| 3.7 Flash | Low | 68.6% | 4.171 | $0.1125 | 25.29s |
| 3.7 Flash | Medium | 68.6% | 4.143 | $0.1188 | 28.75s |
| 3.7 Flash | High | 74.3% | 4.400 | $0.1216 | 28.41s |

Thinking didn't improve quality monotonically. High was best for 3.7, Low was best for 3.6, and Medium and High tied on pass count for 3.5. Another interesting observation is that 3.6-high was actually cheaper and faster than 3.6-medium because despite higher per-turn token usage, it used much fewer tools.

Gemini 3.6 is a good example of how more thinking isn't always better. High thinking passed 21 of 35 tasks, six fewer than Low, and had higher median latency per run. Its lower total spend reflects shorter and fewer agent trajectories, not better task performance.

## Task description and methods

Our primary agent writes the reply a shopper actually receives based on multiple tools and subagent replies. When someone texts "can i return this jacket? also looking for something warmer for fall," that single message pulls on order history, return eligibility, the return action itself, and catalog search (via catalog sub-agent).

The agent has to pick out the right order line, progress the return safely, delegate the product search, hold context across turns, and fold everything into one useful reply. Add customer memory, review requests, follow-up scheduling, and escalation to a person, and a conversation can wander through eight subsystems and five third-party integrations before it's done.

The 35 tasks come from our previous real-customer interaction failures and so represent a real-world "challenge" dataset:

- 11 core single-turn tasks covering returns, exchanges, product questions, mixed return-and-browse requests, and review suppression.
- 12 core multi-turn tasks covering context retention, return execution, follow-on purchases, price deltas, existing labels, and review requests.
- 12 customer-quality regressions covering delivery status, escalation timing, feedback handling, soft fit complaints, and settled conversations.

The 12 multi-turn tasks use an LLM simulator that plays the customer. The simulator gets a persona, an opening text, and a private goal the agent never sees (say, "return the jacket, then get a purchase link for a replacement"). It reacts to whatever the agent actually says, deciding each turn whether to keep going.

Grading has two layers. First, deterministic expectations check the trace for hard facts: were the right tools called, is every product claim grounded in tool evidence, did required artifacts like a purchase link survive into the customer-visible reply, did the agent escalate when it shouldn't have. Second, an LLM judge reads the scenario definition, the simulator's private goal, the full transcript, and compact tool evidence, then scores the conversation 1 to 5 against a fixed rubric. A task passes only if no deterministic expectation fails and the judge scores 4 or 5. A 3 (partially succeeds but misses an important requirement) is counted as a fail.

Costs cover the customer agent and its fixed helper routes, excluding the eval simulator and judge. We priced every usage row at Google's public Standard tier as of August 13, 2026, then divided each configuration's total agent cost by all 35 evaluated runs, whether they passed or failed. Median latency per run gives each run one observation: we sum its customer-agent turn durations, including fixed helper work, then take the median across the 35 runs.

## Takeaway

Gemini 3.7 Flash High matched the quality leaders across the complete 35-task set while running at 52% lower cost per run and with 46% lower median latency per run than our production 3.5 Medium configuration. After two Gemini releases in a row that couldn't beat their predecessors on our benchmarks, this one earned the upgrade.