CityChat is a city guide you talk to. It runs in 16 cities in Poland and Spain, and most conversations are in Polish. In early October the chat ran on gemini-3.1-flash-lite, and Google's deprecations page listed 7 May 2027 as the earliest date it might be switched off, with 3.5 Flash-Lite named as the replacement. Changing the model is one environment variable; instead I spent two days measuring. I ran the tests, picked what to test after each round and made the final call.
What I measured
The harness was the golden suite I already had in CI: 35 tests (45 by the second day) against a small fixture of eight places and two events in a made-up town much like Przemyśl, with the production prompt.
The tests check behavior. Every place an answer mentions has to be one the model was given, a broad evening question in Polish has to mix bars and restaurants, a sponsored place that doesn't fit the question has to be left out, and the model shouldn't ask "near you or in the whole city?" when the user already named the city. Each model ran the suite ten times and the flaky tests 40 to 50 times, streaming, from Poland, with the fallback switched off so every model stood on its own.
Next to the pass rates I counted two more things. One was replies that cite a place id the model was never given. The other was Polish language errors: a second model read the replies as a blind copy editor, without knowing which model wrote them, and then every flag it raised was checked one by one.
On the first day Google's replacement, 3.5 Flash-Lite, passed 96% of the suite (93% before I discounted failures that came from my own language check) against 98% for the model in production, and it was a little faster. On fresh Kraków content it made about 22 Polish errors per 80 replies against about 4, and it wrote "w piwnici", a case ending that doesn't exist, in 16 of 170 Polish replies. It also ignored a vegan user's saved preference in 15 of 40 runs. One line in the prompt fixed that (39 of 40); for the Polish there was no known fix.
Google's other suggestion for new projects, 3.8 Flash, wrote the best Polish of anything I measured, about one error per 80 replies. It also took 2.6 seconds to the first word at the median, more than 10 seconds on 13% of calls, and on 3% it didn't answer within a minute, which is too slow for a chat on a phone.
The fallback I had configured, 3.5 Flash, was worse than the primary on everything I measured: six times the price, and invented place ids in 9 of 420 replies. It was also at the same provider, so a problem at Google would hit both.
So on day one the answer was to keep the current model and pick a successor later, with an A/B on staging.
The second day
The next morning I tried two models from other providers, OpenAI's GPT-6 Luna and Claude Haiku 5.5, which had come out the day before. Luna wrote clean Polish and was the cheapest, but in 18% of replies it asked "near you or in the whole city?" instead of answering. Haiku was the quickest to start and got the sponsored-place rule right every time, but it made about nine times as many Polish errors.
Luna's problem looked like a prompt problem; the rule for when to ask that question had been tuned on Gemini, which almost never asks it. I rewrote the rule so the default is to answer, and the question comes only when the request is broad and gives no hint of where. Unwanted questions fell from 18% to 0.4%.
Luna was still slower to the first word. OpenAI's Fast tier fixed that: 0.62 seconds at the median, the same as Gemini. Fast costs twice the standard price, and even so Luna was about 20% cheaper than Gemini at list price.
One thing the sessions couldn't check was the project's region in OpenAI's dashboard, so I looked myself. The region of an existing project is fixed, and the dialog for a new one didn't offer the EU either; EU residency would need an approval through their sales team, and the Fast tier can't be combined with it anyway. The chat runs from a global project for now, as it did on Gemini's API before.
The final run was about 750 calls per model, on the code that was in production that day:
| Gemini 3.1 Flash-Lite | GPT-6 Luna, Fast | Claude Haiku 5.5, thinking on | |
|---|---|---|---|
| Checks passed | 96.5% | 99.4% | 99.0% |
| Polish errors per 60 replies, each one checked | ≈4 | ≈8 | ≈19 |
| First word, median | 0.63 s | 0.61 s | 1.62 s |
| List price per 1,000 calls | $1.56 | $1.24 | $1.13 |
Gemini's Polish was a bit cleaner, but Luna's mistakes were awkward phrasing and not broken words. Haiku needed thinking switched on to get its structure right - I asked for that second round - and then it took a second longer to start; with the Polish on top of that, it was out. I moved the chat to Luna that evening and kept Gemini as the fallback, so the fallback is now at a different provider.
Before the switch I changed the design of that question as well. A broad question from someone whose location the app knows now gets an answer for the area around them straight away, with distances, and two chips at the end - "near me" and "anywhere in the city" - instead of a question first.
Where the measuring went wrong
My harness favored the model I already had. In a July A/B, 6 of the 8 failures of the challenger came from my own code: the function that decides whether a reply is in Polish used a word list built on the incumbent's replies. This time every failure that hinged on that function was re-scored with a stricter check, and still the language check marked two of Luna's correct replies as failures in the final run.
The spend I tracked during the runs was also about three and a half times lower than what the billing page showed later. The same key served other test runs on those days, so I can't split it cleanly, but the estimate was wrong either way.
Two days can't show drift from one week to the next, and Flash-Lite had drifted before: the same test failed 21 of 80 runs one day and 18 of 20 the next.