Gemini 3.5 Flash-Lite lost to 3.1 Flash-Lite, and more thinking didn't help
We benchmarked Gemini 3.5 Flash-Lite against Gemini 3.1 Flash-Lite and Gemini 3 Flash Preview on an e-commerce catalog agent. The newer model cost more and passed fewer tasks end to end.
Ilya Valmianski 
When Gemini 3.5 Flash Lite came out, we got super excited. A big part of what makes our agent work well is being able to understand a customer’s catalog, which may include thousands of SKUs. For this we delegate to a subagent, historically run on Gemini Flash and Flash-Lite models, that crunches through search results and surfaces the most appropriate items. This often takes multiple search-tool calls, so being cheap per token doesn’t necessarily mean being cheap per task as some models tend to waste a lot of tokens and run extraneous tool calls. Being well-grounded is also very important, as seeing potentially hundreds of related results can cause models to hallucinate combinations that are not part of the catalog.
However, this was the first time a newer model failed to beat an older one. We ran the comparison on the catalog-browsing agent inside our customer support automation for Shopify merchants. We compared gemini-3-flash-preview, gemini-3.1-flash-lite, and gemini-3.5-flash-lite as the model driving that agent, running both Lite models at minimal, medium, and high thinking.
The result: gemini-3.5-flash-lite lost. No 3.5 configuration beat gemini-3.1-flash-lite on the downstream benchmark at any thinking level. The best profile overall was 3.1 at medium thinking, which also beat its own high setting on pass rate, cost, and tokens.
The task
When a shopper texts something like “i am looking for a puffer jacket to match my [recently bought] pants” the customer-facing agent delegates the catalog part of the turn to the catalog subagent, passing a natural-language goal plus whatever structured constraints it extracted: products and variants already in the conversation, order IDs, size, color, category, budget.
On top of that goal, the catalog agent gets a context packet built for the conversation: recent messages, recently referenced products, the customer’s order history, a style profile computed from past purchases (likely sizes, preferred categories and vendors), and a store profile describing the merchant’s catalog (merchandise domain, search rewrite rules, vocabulary, grounding guidance).
It then runs a bounded tool loop over seven domain-specific tools, plus the ability to spawn additional subagents. The tools cover search over a mirrored copy of the merchant’s Shopify catalog, product detail lookups, live price and availability validation, and on-demand context like recent orders, style profile, merchant guidance, etc.
Each turn ends in a structured result: short recommendation text, grounded options and alternatives (product cards carrying price, availability, image, and purchase URL), and full product details for follow-up questions. A deterministic validator checks every product, variant, price, availability, and link claim in that result against the tool transcript. Anything unsupported is flagged as a grounding error.
Downstream, the customer-facing agent treats this result as evidence and merges the grounded cards and recommendation text with the non-catalog parts of the turn and writes the reply the shopper actually receives. The models below were swapped in as the catalog agent only; everything around it stayed fixed.
How we measured
We collected a number of examples that have historically caused our agent to make mistakes, across two test types:
- 13 isolated tasks that call the catalog agent directly.
- 42 production-shaped conversations in which a separate customer-facing agent receives the catalog result and writes the reply.
Beyond objective measures, we also asked the LLM to subjectively judge the quality of the response again a rubric. In every downstream run, the customer-facing agent, simulated customer, and judge were pinned to gemini-3.5-flash.
Costs below cover the catalog agent only, at standard Vertex AI provider rates. Every per-pass metric keeps failed work in the numerator: total catalog spend, tokens, tool calls, or wall time divided by successful tasks.
Isolated catalog tasks
Isolated benchmark results
Detailed results across 13 direct catalog-agent tasks, grouped by model family.
| Model family | Thinking | Pass rate | Avg. judge | Cost / pass | Tokens / pass | Tool calls / pass | Seconds / pass | Grounding errors |
|---|---|---|---|---|---|---|---|---|
| 3 Flash Preview | Preview | 92.3% | 4.692 | $0.0233 | 41.3k | 5.42 | 4.85 | 4 |
| 3.1 Flash-Lite | Minimal | 84.6% | 4.692 | $0.0104 | 36.2k | 5.00 | 6.77 | 2 |
| Medium Overall winner | 100.0% | 4.923 | $0.0087 | 29.8k | 4.31 | 2.70 | 0 | |
| High | 92.3% | 4.769 | $0.1143 | 116.3k | 5.83 | 37.07 | 6 | |
| 3.5 Flash-Lite | Minimal | 84.6% | 4.462 | $0.0147 | 37.9k | 5.55 | 6.62 | 0 |
| Medium | 100.0% | 5.000 | $0.0179 | 37.5k | 5.23 | 5.47 | 1 | |
| High | 100.0% | 4.923 | $0.0230 | 40.3k | 5.46 | 13.89 | 1 |
Both Lite models solved all 13 tasks at medium thinking, but 3.1 Flash-Lite got there with the lowest cost, fewest tokens, fewest tool calls, and shortest wall time per pass.
High thinking moved the two generations in opposite directions. It kept 3.5 Flash-Lite at 13/13, at more time and spend. On 3.1 Flash-Lite it slipped to 12/13 while burning 1.40 million catalog tokens (more than 3.5 times its medium run) and throwing 6 grounding errors, the most of any profile.
Downstream agentic tasks
The direct benchmark doesn’t test the handoff. A catalog result can appear all right in isolation but the main agent might still choose to make subsequent requests with alternative plain-English queries before replying to the customer. Because of this we also evaluate catalog-browsing in the context of full agent turn.
Downstream benchmark results
End-to-end pass rate after the catalog output flowed through the customer-facing agent, grouped by model family.
| Model family | Thinking | Pass rate | Avg. judge | Cost / pass | Tokens / pass | Tool calls / pass |
|---|---|---|---|---|---|---|
| 3 Flash Preview | Preview | 91.4% | 4.800 | $0.0547 | 112.1k | 6.81 |
| 3.1 Flash-Lite | Minimal | 94.3% | 4.857 | $0.0221 | 91.2k | 5.73 |
| Medium Overall winner | 97.1% | 4.943 | $0.0219 | 88.4k | 5.44 | |
| High | 88.6% | 4.686 | $0.0937 | 152.0k | 6.42 | |
| 3.5 Flash-Lite | Minimal | 91.4% | 4.771 | $0.0343 | 104.5k | 6.56 |
| Medium | 88.6% | 4.686 | $0.0422 | 115.7k | 6.74 | |
| High | 91.4% | 4.714 | $0.0522 | 121.8k | 6.94 |
While in isolated test 3.5 Flash Lite medium and high reached the same 100% pass rate as 3.1 Flash Lite, in end-to-end testing 3.1 Flash Lite was a clear winner.
3.1 Flash-Lite medium passed 97.1% downstream tasks, the best pass rate in the table. It beat every 3.5 Flash-Lite setting and its own high setting on pass rate, cost per pass, tokens, and tool calls, all at once. Against 3 Flash Preview it ran roughly 60% cheaper per passing task, on 21% fewer tokens and 20% fewer tool calls.
What we took away
The newer model lost. 3.5 Flash-Lite scored slightly higher on isolated judge scores and stayed at 13/13 even at high thinking, but downstream, every 3.5 configuration passed fewer tasks than 3.1 at minimal or medium and cost at least 50% more per passing task than 3.1 medium. Overall, it’s an interesting reminder: if we had upgraded based on the release notes, this agent would have gotten worse and more expensive.
