LLM regression testing: a one-week routine for model upgrades

LLM regression testing for a model upgrade means running a frozen set of your own production tasks through the old and new models and comparing the results item by item. As a routine, it takes about a week. Build a golden dataset from logs and past incidents. Check outputs in code, and bring in a calibrated LLM judge only for what code can’t check. Price the switch per task at the rate you’ll pay after any launch offer, and measure p95 latency at your own load. Read every regression yourself, then shadow live traffic and run a canary behind a switch you can flip back.

Models now change monthly. Anthropic released Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5 between 22 September and 7 October 2026, each with notes on code that can break.¹ Google made Gemini 4 Argon generally available on 13 October at an introductory $2 per million input tokens and $10 per million output tokens, rising to $4 and $20 on 1 December.²³

We haven’t tested Argon or any other model for this post. Vendor claims are labelled, and prices are as of 21 October 2026.

Why leaderboard gains don’t carry over to your product

A leaderboard scores a model on someone else’s tasks, harness and prompts, and small changes to the harness can reorder the top. Google’s launch post says Argon ranks first on the Vals Index of finance, coding, legal and tax tasks.² It did on 13 October, at 68.90% to Claude Sonnet 5.5’s 67.04%.⁴ Six days later, version 2.2 of the index, with updated component tests, put Sonnet 5.5 first at 70.85%, Claude Opus 5.5 second at 70.18% and Argon third at 69.24%.⁵ Vals adds that Argon would lead if answers from a fallback model after a refusal counted as failures. The top three are 1.6 points apart, and their order depends on how refusals are scored.

Your prompts were also tuned to the old model’s habits, and vendor guides say those habits change between generations.

• Google says Argon “can spend more of the limit on thinking than earlier models” and ignores temperature, top_p and top_k. It asks you to run your evaluations at all three thinking levels.⁶
• Anthropic says effort level names “don’t correspond to the same amount of thinking across models”, and that instructions tuned for Opus 5 may no longer be needed on Opus 5.5.⁷⁸
• OpenAI says GPT-6 Astra is stronger at following instructions than its previous models, and can be more sensitive to instructions in skill files and AGENTS.md.⁹

Google and Anthropic both count thinking against the output limit, so a cap sized for a model that thought less can cut an answer off halfway through its JSON.⁶⁷

Build a golden dataset from production traffic

The test set should look like your traffic, plus the cases that have hurt you before.

• Sample by intent. Draw recent requests from each intent or route in proportion to traffic, with a floor for rare intents that matter, such as refunds.
• Add hard cases. Include long inputs, every language you serve, ambiguous requests, requests that should be refused, and items the old model gets right only sometimes.
• Add every past incident as a permanent item.
• Freeze it. Store each item as the model saw it, with the prompt version, retrieved passages and tool results, so a replay changes only the model. For agents, replay recorded tool responses or use a sandbox.
• Label expected outcomes. Use an exact value where one exists, required facts and forbidden claims where it doesn’t, and flags for items that should be refused or should end with a question.
• Scrub personal data under your own policy first.

Run both models on every item with the same prompts and settings, apart from changes the migration guide requires. Evan Miller’s “Adding Error Bars to Evals” recommends analysing these question-level paired differences, which he calls a “free” reduction in variance, and repeating questions whose answers vary.¹⁰ His worked example needs about 1,000 questions to detect a 3-point difference. Our post on measuring model drift works through the arithmetic.

Checks in code first, an LLM-as-a-judge second

Anthropic’s evaluation guide calls code-based grading the “fastest and most reliable” method.¹¹ It also catches the mechanical changes above on every item.

• Schema. Validate every structured answer against its schema, and count truncated JSON separately.
• Tool calls. Check that required calls happen with valid names and arguments, and that nothing is called when nothing is needed. In unattended flows, count turns that end in a question or a status note.
• Citations. Every cited source must exist in the retrieved set.
• Refusals. Count them per intent. On Claude, a classifier decline is a normal response with the stop reason “refusal”.⁷
• Length and format. Log visible and thinking tokens, and flag Markdown where your interface shows plain text.

In 9io Alpha, our research platform, code checks the allowed values, ranges and required fields in the judge model’s answer on every research verdict before anyone sees it. Check your test judge’s output the same way.

Use the judge only for what code can’t check, such as correctness or tone. First label a sample by hand and measure the judge’s agreement with a statistic that corrects for chance. Thakur and colleagues found that judges with high percent agreement could still give scores far from human ones, and that judges leaned towards leniency.¹² A lenient judge can pass both models and hide the regression you’re looking for.

Zheng and colleagues measured several judge biases on 2023 models, and each can tilt a comparison between two model generations.¹³

• Verbosity. On 23 answers, padding each with a rephrased copy of its own list fooled Claude-v1 and GPT-3.5 in 91.3% of cases and GPT-4 in 8.7%.
• Order. GPT-4 gave the same verdict with the answers swapped in 65% of cases, so judge each pair both ways.
• Reference answers. On 10 maths questions judged in both orders, GPT-4 passed a wrong answer 14 times in 20 with the default prompt and 3 times when given a reference answer.
• Self-preference. Panickssery and colleagues found that models can recognise their own outputs, and that stronger self-recognition went with stronger self-preference.¹⁴ Use a judge from a third model family, and pin its ID too.

Cost per task, at the price after the introductory period

Compare cost per successful task on the golden set. Multiply input and output tokens by their prices, counting thinking as output, multiply by one plus the retry rate, and divide by the pass rate.

• Thinking. Google bills Argon’s output “including thinking tokens”, and Anthropic bills thinking as output “even when the thinking text is not returned to you”.³⁸
• Length. OpenAI says GPT-6 Astra “tends toward detailed, formatted responses”, and also reports, as a vendor claim, a lower estimated cost per task than earlier models in several of its evaluations.⁹
• Cache hits. Argon supports only implicit caching, with a 95% discount on hits for prompts of at least 4,096 tokens.⁶ On Opus 5.5, changing the effort level between requests invalidates the cache.⁷
• Tokenizers. Anthropic says its tokenizer since Opus 4.7 may use roughly 1x to 1.35x as many tokens for the same text as earlier models, varying by content.⁸ Count your own prompts with each model’s counter.

As of 21 October 2026, Argon costs $2 per million input tokens and $10 per million output tokens through 30 November 2026, then $4 and $20 from 1 December, with cached input going from $0.10 to $0.20.³² Price the decision at the later rate. Other models’ prices are in our LLM API price comparison of 8 October.

Latency and rate limits on a newly launched model

Measure p50 and p95 end to end, time to first token, and time to the first token a user sees, which thinking delays. Do it from your region at your concurrency, and again on live traffic during the shadow run.

Google says hard Argon prompts “can take several minutes”, recommends HTTP client timeouts of at least 15 minutes, and lists its high thinking level as having the “slowest first token”.⁶

Argon isn’t on the free tier, and Google doesn’t publish its per-minute limits. Each project sees its own in AI Studio, Google says they “are not guaranteed”, and spend-based limits may apply over a rolling 10 minutes, from $10 at Tier 1 to $200 at Tier 3, depending on billing history.¹⁵ At Tier 1 and Argon’s introductory output price, $10 covers about one million output tokens every 10 minutes, thinking included. Anthropic gives Opus 5.5 its own limit, outside the bucket shared by Opus 4.5 to 4.8, so headroom on an older model doesn’t follow you.¹⁶ Load-test at peak concurrency and count 429 errors and timeouts as failures.

Behaviour changes to read by eye

Read every item the old model passed and the new one failed, plus a sample that both passed, side by side by intent and language.

• Tone and length. Google says Argon “answers questions more directly with less hedging” and follows style instructions better than its earlier models.² OpenAI says GPT-6 Astra tends to use lists, tables and Markdown and may reuse phrases across sessions.⁹ An instruction added to stop the old model hedging can over-correct.
• Stopping to ask. Google trained Argon to “pause for human confirmation before taking irreversible steps”.² OpenAI says Astra is more likely to ask a question when more input could change the result, which “can cause it to stop when the user may expect it to make reasonable assumptions and persist”.⁹ Anthropic says some Opus 5.5 progress updates end the turn without a tool call, which stops an unattended agent loop that reads them as the end of the task.⁷
• Refusals. Google says Argon is “significantly better at not fulfilling harmful requests than our previous models”, and Anthropic says Opus 5.5’s classifiers cover more categories than Opus 5’s.²⁸ Test legitimate requests about security and biology.
• Tool plumbing. Opus 5.5 and Sonnet 5.5 reject forced tool choice. Opus 5.5’s notes between tool calls arrive in thinking blocks, so an app that streamed them “goes quiet between tool calls”.¹⁸
• Languages. Read samples in each language you serve, and compare token counts per language.

Shadow testing, a canary and pinned versions

In shadow testing, each live request is copied to the new model, and its answer is checked by the same code, logged and discarded, so users never see it. It costs money and sends user content to another model. If you switch LLM provider, check that your data processing terms cover that.

Then run a canary. Google’s SRE workbook defines canarying as “a partial and time-limited deployment of a change in a service and its evaluation”, with the rest of the service as the control.¹⁷ Send a small share of traffic to the new model, compare its metrics with the control, and keep the model choice in a runtime flag so rollback is a config change.

Pin the model you tested. What stays fixed differs by vendor, as of 21 October 2026.^181920212223

Pinning has limits. Anthropic says changes to its serving infrastructure can alter behaviour slightly on an unchanged ID.¹⁸ Since 8 October, Google’s deprecated gemini-3.7-flash has routed requests to gemini-3.8-flash, an upgrade its callers didn’t choose.²⁴ Claude’s retirement dates can also differ on Amazon Bedrock and Google Cloud, which set their own.¹⁹ Watch each model deprecation page, poll the shutdown_date field in OpenAI’s Models API, and keep the golden set running on a schedule so a forced LLM model upgrade gets the same test as a chosen one.

LLM regression testing checks, pass rules and blockers

Set tolerances before you run. The table assumes 2 points for task success; use your own.

A paired bootstrap for old versus new in Python

The function returns the change in pass rate from old to new with a 95% interval, resampling items with replacement as in Koehn’s paired bootstrap of 2004.²⁵ Pass each item’s intent as strata to keep the set’s mix of intents in every resample.

import random
from collections import defaultdict


def paired_bootstrap(old, new, strata=None, n_boot=10_000, seed=0):
    """Change in pass rate (new minus old) on the same items, with a 95% interval.

    old, new: 0/1 or True/False results, one per item, in the same order.
    strata: optional label per item (an intent, say); resampling then keeps the mix.
    Returns (difference, low, high, regressions, fixes).
    """
    if not old or len(old) != len(new):
        raise ValueError("need one old and one new result for every item")
    deltas = [int(b) - int(a) for a, b in zip(old, new)]  # -1 regressed, +1 fixed
    groups = defaultdict(list)
    for label, d in zip(strata or [None] * len(deltas), deltas):
        groups[label].append(d)
    rng = random.Random(seed)
    n = len(deltas)
    boots = sorted(
        sum(sum(rng.choices(g, k=len(g))) for g in groups.values()) / n
        for _ in range(n_boot)
    )
    low, high = boots[int(0.025 * n_boot)], boots[int(0.975 * n_boot) - 1]
    return sum(deltas) / n, low, high, deltas.count(-1), deltas.count(1)

Read the interval first. Only items where the two models disagree change the difference, and their share sets the interval’s width. If the interval is wider than the regression you care about, the set is too small. With a tolerance agreed in advance:

• Switch on this check when low is above minus the tolerance.
• Block when high is below zero.
• Otherwise add items, or stay on the old model.

For a plain pass rate over hundreds of items, Miller’s analytic formula gives much the same interval, and he regards a bootstrap as unnecessary unless the sampling scheme or estimator is complicated.¹⁰ A stratified sample is such a case, and so is cost per successful task.

LLM evaluation checklist for a model upgrade

Days 1 and 2

• Sample production requests by intent, add hard cases and past incidents, scrub personal data and freeze the set.
• Label expected outcomes, including refusals and clarifying questions.
• Pin both model IDs and apply only the changes the migration guide requires.

Days 3 and 4

• Run both models on every item, repeating where answers vary, then the code checks.
• Calibrate a judge from a third model family, with reference answers and both orders.
• Compute the paired difference and interval, and read every regression.

Day 5

• Price each successful task at the post-introductory rate, thinking and retries included.
• Measure p50, p95 and first visible token at peak concurrency, and count 429 errors.

Then

• Shadow live traffic, then canary behind a runtime flag with a tested rollback.
• Note the retirement date, and rerun the golden set on a schedule.

Frequently asked questions