AI Model Swapping: How to Check Which Model You Get

9 min read · Last updated: October 24, 2026

AI model swapping is when an app or API router answers your request with a different model, a cheaper host, or a lower-precision copy of the model you picked. Usually you never see it happen. You can catch it by logging which provider served each request, pinning providers and precision, and running a fixed test set against the official API.

Came here from Instagram, LinkedIn, X or YouTube? Here is the full source list you came for. Everything below is built from these, nothing else.

The post that started this

• The Cloud Token Swapping Scam, Medium, Oct 24 2026 (behind Medium's wall; I could only read its summary, so nothing below is taken from it)

The incidents

• Anthropic: A postmortem of three recent issues, September 2025
• Simon Willison on Sam Altman's GPT-5 rollout update, Aug 8 2025
• VentureBeat: OpenAI returns old models after a bumpy GPT-5 rollout, Aug 8 2025

The measurements

• Moonshot AI: Kimi Vendor Verifier, released with Kimi K2.6
• K2-Vendor-Verifier results on GitHub, tested Nov 15 2025
• OpenRouter: Provider Variance, Introducing Exacto, Oct 21 2025

The docs

• OpenRouter: Provider Routing
• OpenRouter: Auto Exacto
• OpenRouter: Get a generation (API reference)

Key takeaways

• The same open model, run by different hosts, is not the same product. In Moonshot AI's November 2025 test, tool-call schema accuracy for Kimi K2 0905 ranged from 100% on the official API down to 71.96% on one third-party host.
• Routers pick for you by default. OpenRouter's default routing prioritizes price, weighting the cheapest stable providers by the inverse square of price.
• Big labs get it wrong too. A routing bug sent 16% of Claude Sonnet 4 requests to the wrong servers at its worst hour on August 31, 2025.
• Most silent downgrades in the public record are bugs, wrong settings, or cost-first routing. The evidence for deliberate fraud is thin. The effect on your app is the same either way.
• You can check it. Log the serving provider on every request, pin precision and fallbacks, and rerun a fixed test set of about 4,000 requests or fewer against the official API.

What is AI model swapping?

AI model swapping means the model that answers your prompt is not the exact model you think you chose. It can be a different model picked by a router, the right model served by a cheaper host, or the right weights run at lower precision or with wrong settings. The name on the dashboard stays the same.

Think about taking your car to a garage and asking for a specific brand of brake pads.

The mechanic fits a cheaper pad that looks the same. The car drives out fine. It stops fine in the car park. You only find out on a wet motorway at speed, when the stopping distance is longer than it should be.

LLM swapping works the same way. A cheap or quantized model answers "what is a JWT" perfectly. It falls apart on the hard parts: long outputs, strict JSON, tool calls, multi-step reasoning. Those are exactly the parts your production app depends on.

There are three versions of the swap, and they need different checks:

Type of swap	What changes	Typical cause	How you notice
Model swap	A different model answers	A router or "auto" mode picks a cheaper model	Response quality drops on hard prompts; the logged model name differs
Provider swap	Same model name, different host	Price-first routing or fallbacks after an outage	The serving provider in logs changes between requests
Precision or config swap	Same model, same name, lower fidelity	Quantization (for example int4 instead of bf16), wrong decoding parameters, inference bugs	Tool calls fail schema checks; long answers degrade

How does model routing actually work?

Model routing is the step between your request and the GPU that runs it. A router reads your request, looks at which models and providers can serve it, and picks one based on rules: price, speed, uptime, or a guess at task difficulty. If the first choice fails, a fallback rule can send the request somewhere else.

Here is the path a single request takes through a typical aggregator, step by step:

1. Your app sends a request naming a model, for example an open-weight model like Kimi K2, the large language model from Moonshot AI.
2. The router lists every endpoint that hosts that model. For popular open models there can be ten or more hosts.
3. It filters out unhealthy hosts. OpenRouter's docs say providers without significant outages in the last 30 seconds are considered first.
4. It picks by price unless told otherwise. OpenRouter's documented default is to "load balance requests across providers, prioritizing price," choosing among the cheapest stable providers weighted by the inverse square of price.
5. It falls back if the pick fails. allow_fallbacks defaults to true, so the request moves to a backup provider when the primary is unavailable.
6. The host runs the model its own way. Precision (fp8, int4, bf16), the inference engine, and default sampling settings are the host's choice, not yours.
7. You get an answer with the model name you asked for.

None of those steps is a scam. Price-first routing is a reasonable default for chat. The problem starts when you need a guarantee and you never asked the router for one.

Consumer chat apps do a version of step 4 at the model level. When OpenAI launched GPT-5 in August 2025, ChatGPT used an "autoswitcher" to decide which GPT-5 variant answered each prompt. On August 8, 2025, Sam Altman said it was "out of commission for a chunk of the day," which made GPT-5 seem "way dumber," and promised to be "more transparent about which model is answering a given query."

That is the honest version of this problem. A router broke, the CEO said so, and users had felt it before anyone told them.

Why does the same model behave differently on different providers?

The same open-weight model can score very differently across hosts because each host chooses its own quantization, inference engine, parsing code and default parameters. Moonshot AI tested this directly with its own model and found a "stark contrast" between third-party APIs and the official API, a gap it called widespread.

The best public data here comes from the model maker itself. Moonshot open-sourced the K2-Vendor-Verifier, which sends about 4,000 tool-calling requests to each provider and compares the results with Moonshot's official API. It measures two things: whether the model decides to call a tool when the official API would (trigger similarity, as an F1 score), and whether the tool call matches the JSON schema it was given (schema accuracy).

Here are selected results for Kimi K2 0905, tested November 15, 2025, at temperature 0.6:

Endpoint	Tool-call responses	Schema errors	Schema accuracy
Moonshot AI (official)	1,274	0	100.00%
Fireworks	1,453	0	100.00%
DeepInfra	1,365	0	100.00%
SiliconFlow	1,305	3	99.77%
Chutes	1,271	42	96.70%
Baseten	1,243	342	72.49%
Together	1,266	355	71.96%

Read the bottom two rows as a developer. More than one in four tool calls came back broken. If your agent depends on those calls, it fails roughly a quarter of the time, with the same model name on the invoice.

Two more details from the same repo matter:

• Passing the schema is not enough. Groq scored 100% schema accuracy on K2 0905 but only 69.52% trigger similarity, below the 80% bar Moonshot treats as acceptable. Clean JSON, but the model called tools at a different rate than the official API.
• Results move. In the K2-thinking run on the same date, DeepInfra dropped to 86.91% schema accuracy, while it scored 100% on K2 0905. A host that is good for one model version can be weak on the next.

Moonshot's write-up names the causes. Early anomalies were mostly misused decoding parameters, so it now enforces temperature 1.0 and top_p 0.95 in Thinking mode at the API level. Its long-output AIME2025 test exists to catch quantization damage and KV cache bugs that short benchmarks miss.

OpenRouter reached the same conclusion from live traffic. In October 2025 it wrote that tool-use propensity and call accuracy "vary between providers significantly more than the standard benchmarking would suggest," and launched Exacto endpoints that route tool calls to hosts with better measured success.

Is it a scam, or just bad infrastructure?

In the public record, most silent downgrades are bugs, misconfiguration or cost-first defaults, not proven fraud. That does not make them harmless. A model that quietly gets worse costs you the same whether someone planned it or not, so treat verification as your job, not the vendor's.

The clearest documented case is Anthropic's own. In its September 2025 postmortem, Anthropic described three overlapping bugs:

• A routing error. From August 5, 2025, some Claude Sonnet 4 requests went to servers set up for the upcoming 1M-token context window. It started at 0.8% of requests. A load balancing change on August 29 made it worse, and at the worst hour on August 31, 16% of Sonnet 4 requests were affected.
• Sticky routing made it personal. Follow-up requests tended to land on the same wrong server, so some users got hit again and again. About 30% of Claude Code users active in that period had at least one message misrouted.
• Two token-generation bugs. A TPU server misconfiguration put Thai or Chinese characters into English replies, and a compiler bug in approximate top-k sometimes dropped the most likely token.

Anthropic also stated: "We never reduce model quality due to demand, time of day, or server load."

You can take that statement at face value and still learn the lesson. The users who noticed first were the ones watching their outputs closely. If a lab with its own inference team can serve the wrong thing for weeks, a budget app reselling five providers can too.

One more caution. I could only read the summary of the Medium post that inspired this article, so I have not repeated any of its specific claims about particular apps. If you see a screenshot claiming a named app is "swapping tokens," ask for the test set and the logs before you believe it.

When should you let a router choose, and when should you pin?

Let a router choose when the work is cheap, forgiving and easy to check by eye. Pin the provider, the precision and the fallbacks when you depend on tool calls, strict JSON, long outputs, or benchmark-level reasoning. The rule is simple: the more a wrong answer costs you, the less choice you give the router.

Use case	Let the router pick	Pin it down
Casual chat, brainstorming	Yes, price-first is fine	Not worth it
Summaries you read yourself	Yes	Optional
Agents that call tools	No	Pin provider, require parameters, turn off fallbacks
Structured JSON into a database	No	Pin provider and precision
Evals or benchmarks you publish	No	Use the official API or a verified host only
Long reasoning or code generation	Risky	Pin precision; avoid low-bit quantization

On OpenRouter, the controls are documented in the provider object:

• allow_fallbacks: false stops the silent jump to a backup host. The trade-off is that requests can fail outright instead.
• require_parameters: true skips providers that do not support every parameter you sent, instead of letting them ignore it.
• quantizations filters by precision. Accepted values include int4, int8, fp8, fp16, bf16 and unknown.
• order or sort replaces price-weighted load balancing with your own ordering.
• data_collection: "deny" limits routing to providers that do not store or train on your data. The default is allow.

Tool-calling requests on OpenRouter now go through Auto Exacto by default, which reorders hosts by measured tool-calling success. Setting sort to "price" or adding :floor to the model slug turns that off, so do not add either one by accident.

One limit to keep in mind: the quantizations filter matches what each provider says it runs. It is a promise, not a measurement. That is why the last step is testing.

How can you check which model an AI app is really using?

Check three things on every request: which provider served it, at what settings, and whether its output still matches a known-good baseline. Log the first two automatically. Test the third on a schedule with a fixed prompt set, compared against the official API of the model maker, the way Moonshot's verifier does.

Here is a small Python example against OpenRouter. It pins the request, then asks the generation endpoint who actually served it.

import os
import time
import requests

BASE = "https://openrouter.ai/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"}

body = {
    "model": "moonshotai/kimi-k2-0905",
    "messages": [{"role": "user", "content": "Reply with the single word OK."}],
    "provider": {
        "allow_fallbacks": False,
        "require_parameters": True,
        "quantizations": ["fp8", "bf16"],
        "data_collection": "deny",
    },
}

reply = requests.post(f"{BASE}/chat/completions", headers=HEADERS, json=body).json()
gen_id = reply["id"]

time.sleep(2)
stats = requests.get(f"{BASE}/generation", headers=HEADERS, params={"id": gen_id}).json()["data"]

print("model:   ", stats["model"])
print("provider:", stats["provider_name"])
print("tokens:  ", stats["native_tokens_prompt"], "in /", stats["native_tokens_completion"], "out")
print("cost:    ", stats["total_cost"], "USD")

The generation endpoint returns provider_name, model, total_cost and the token counts as reported by the provider. Store those four fields next to every request in your own logs. If provider_name starts changing while your code did not, routing changed under you.

Then add a canary test. In words, it looks like this:

1. Freeze a test set. Take 50 to 200 real prompts from your app, including tool calls with strict JSON schemas and a few long outputs. Keep the expected answers.
2. Get a baseline from the model maker's official API at the recommended settings.
3. Run the same set through the app or router you are checking, at the same temperature.
4. Score two numbers, the same ones Moonshot uses: how often a tool call is triggered compared with the baseline, and the share of tool calls that pass schema validation.
5. Alert on drift. A drop from 100% to 96% schema accuracy is worth a look. A drop to 72% is a different product.
6. Rerun weekly and after every model release, because host results change between versions.

The schema check is short:

import json
from jsonschema import validate, ValidationError

def schema_accuracy(tool_calls, schemas):
    passed = 0
    for call in tool_calls:
        try:
            args = json.loads(call["function"]["arguments"])
            validate(args, schemas[call["function"]["name"]])
            passed += 1
        except (KeyError, json.JSONDecodeError, ValidationError):
            pass
    return passed / len(tool_calls) if tool_calls else 1.0

This mirrors what OpenRouter says it checks on every tool call: valid JSON, a tool name that was in the request, and arguments that match the schema.

You do not need Moonshot's full setup to start. Its full verifier took about 15 hours on two 8-GPU NVIDIA H20 servers. A 100-prompt canary costs a few cents and catches the 72% case on the first run.

If you are trying a cheap reseller for a frontier model, like the ones in our GPT-6 Astra on Kie AI guide, run this canary before you build on it. The same goes for price changes that quietly reroute traffic, which we covered in the DeepSeek V4 Flash pricing and V4 Pro reroute breakdown.

What should budget AI app users do?

If you use an AI app rather than an API, you cannot read routing logs, so test the outputs. Keep five hard prompts you know the right answers to, run them when you sign up and again every few weeks, and compare. If the app will not say which model and host answer you, treat its model label as marketing.

A few practical signs to watch for:

• The app offers a frontier model at a price far below the model maker's own API, with no explanation of how.
• Answers get noticeably shorter or sloppier at busy times.
• Code or JSON that worked last week now fails in small ways, like a missing field or a broken bracket.
• Replies in English pick up random characters from other scripts, the exact symptom Anthropic described in its TPU bug.

None of these proves a swap. Together, they are a reason to run your canary set. And whatever model you end up with, still review AI-generated code properly, because even the right model is wrong sometimes.

FAQ

Can an AI app secretly use a different model than it claims?

Yes, technically nothing stops it. You only see the text that comes back, not the weights that produced it. Routers can pick other models, resellers can choose cheaper hosts, and hosts can run quantized copies. The only defence is to log the serving provider where you can and test outputs against a known baseline.

Does quantization always make a model worse?

Not always, and not equally. Moonshot built its long-output AIME2025 test specifically to catch quantization damage that short benchmarks miss, which tells you the loss often hides in long or complex tasks. Test the precision you are given on your own workload instead of trusting a label.

How do I see which provider served my OpenRouter request?

Call GET https://openrouter.ai/api/v1/generation?id=<generation id> with your API key. The response includes provider_name, model, total_cost and native token counts. Logging these for every request shows you when routing changes.

Do big labs ever downgrade models under load?

Anthropic says it never reduces model quality due to demand, time of day or server load. It did admit that bugs in August and September 2025 degraded some responses, including a routing error that hit 16% of Claude Sonnet 4 requests at its worst hour. Bugs can look like downgrades from the outside.

Pick your five hardest prompts today, run them against the official API and against whatever app you pay for, and save both answers. Then grab the free tools on Art of Code, and if you are building agents on a tight budget, read our guide on building a chatbot that converts without burning budget.

Written by Adithya Guttha, Founder of Art of Code.