None of my products names an AI model in its code. They ask for a job, like summarise or translate, and one small gateway decides who does it.
I did this because model names were rotting all over my projects. Each one also had its own retries, its own keys and its own idea of a timeout. When someone asked what AI cost last month, nobody could say.
Here is what that looks like. Say a help chat needs a summary of a long thread. It asks for summarise and says it will wait 8 seconds. It doesn't know which model answers. If the first one is down, the second one answers and the chat never finds out. The cost lands in one table with the project's name on it.
This note is how to build that: the table, the code, what happens to a call step by step, and what broke.
The table
Each role has a list of models to try in order, and a ceiling on how much it may write. My real table has roles like summarise, classify, extract, translate, illustrate and edit-image.
const ROLES = {
summarise: { chain: ['cheap-fast', 'cheap-backup'], maxOutputTokens: 800 },
translate: { chain: ['good-at-czech', 'cheap-fast'], maxOutputTokens: 4000 }
};
Changing a default model is one line. A project can override a role for itself, and that choice is stored in the database and checked before it is saved. A changed role takes up to three minutes to reach every server, because the lookup is cached for that long.
The ceilings differ a lot. classify may write 200 tokens. The role that writes whole articles may write 16,000. Each role also has a ceiling on how long a caller may ask to wait: 30, 120 or 180 seconds.
The list a call really walks is the project's stored chain with the role's default chain added on the end. That addition is on purpose. A provider can retire every model a project picked, and the call still has somewhere to go.
Reading that list can fail too, when the database is down or a migration is running. Then the default from the code is used and the call carries on, because an AI feature with someone waiting shouldn't die over a settings table. The empty result is cached as well, so a database that is down doesn't get called again on every request.
A role can also carry settings for its models. One of mine runs a thinking model with thinking switched off, because when it was on, 43 of 60 answers were cut off before they said anything. Off, it answers in 0.34 seconds.
The caller
The caller gives three things: the role, the messages and how long it is willing to wait.
const summary = await ai('summarise', messages, 8000, send);
The time limit is required. An AI call with no limit is how a page hangs for a minute. An unknown role throws an error instead of quietly picking something, because a typo that silently uses an expensive model is the worst kind of bug.
The gateway
This is the core of it, and it runs:
function isRetryable(status) {
return status === undefined || status === 408 || status === 429 || status >= 500;
}
function waitMs(status, attempt) {
return (status === 429 ? 2000 : 250) * 2 ** attempt;
}
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function ai(role, messages, timeoutMs, send) {
const config = ROLES[role];
if (!config) throw new Error(`Unknown AI role: ${role}`);
if (!timeoutMs) throw new Error(`A time limit is required for role ${role}`);
const deadline = Date.now() + timeoutMs * 3;
for (const model of config.chain) {
for (let attempt = 0; attempt < 2; attempt++) {
if (Date.now() > deadline) throw new Error('Out of time');
try {
return await send(model, messages, timeoutMs, config.maxOutputTokens);
} catch (err) {
if (!isRetryable(err.status)) break;
if (attempt === 0) await sleep(waitMs(err.status, attempt));
}
}
}
throw new Error(`Every model failed for role ${role}`);
}
send is whatever talks to your provider. Each model gets two attempts, then the next one takes over. A failure that a retry can't fix, like a 404, skips straight to the next model. My longest chain has four models. Mine waits longer after a 429, because a rate limit needs more time than a blip does.
The total budget defaults to three times the per-call limit. Without it, a chain of four slow models turns an 8-second limit into a minute.
One retry layer, one deadline
Retries are the easiest thing here to get wrong, so the rule is that exactly one place does them. The provider's SDK has its own retries switched off, and callers aren't allowed to add any. In the project this design came from, three layers of retries stacked up and turned a 120-second timeout into a 42-minute call. The layer in the middle logged nothing, so it took a while to see.
The deadline is worked out once, when the call starts, and every attempt on every model shares it. If you work it out again per attempt, your total budget quietly becomes a multiple of itself.
A wait never runs past the deadline either. If a provider sends a Retry-After that doesn't fit in the time that's left, the gateway doesn't sit through it. It moves on to the next model, because sleeping until the deadline would mean never reaching the fallback.
What happens to one call
The sample above is the middle of it. In production a call goes through these steps:
- Who is asking. The key is checked, then whether that key may call AI at all, and whether the project's plan includes it. Then the request body is checked.
- Is there money left. The project's spending is compared with its allowance. If it's over, the caller gets a 429 and nothing is spent.
- Which models. The project's own choice for the role, or the default. Retired models and providers with no key are dropped from the list.
- Try them in order. Every failure goes into one of three groups, below.
- Write down the cost. One row in one table.
Step 3 also drops any model that can't do what this request needs: tools, JSON output, reading a picture or a PDF. If nothing is left, the call is refused before any provider is asked, so a request no model can serve costs nothing.
The three groups are the part I'd copy first:
| What happened | Examples | What the gateway does |
|---|---|---|
| A blip | No answer, 408, 429, 5xx | Wait, try the same model again |
| This model can't do it | 401, 402, 403, 404, quota gone, prompt too long, unknown model, empty answer | Go to the next model |
| Anything else | A refused prompt, a bad request | Stop and tell the caller |
Read that one from the top. The first model answers 429, so the gateway waits and asks it again. The second answer is a 404, which no amount of waiting will fix, so the next model gets the call and answers. The caller only sees the answer.
An empty answer goes in the middle group, and you still pay for it. So does a role that asked for JSON and got something that doesn't parse. That one gets one more try on the same model, inside the same deadline, and both attempts are billed. If the second one fails too, the caller gets an error, never text it can't use.
Empty answers are less rare than they sound. One cheap thinking model spent a whole 8,000-token allowance on thinking and returned nothing, 12 times out of 12. If I'd counted that as a success, every caller would have received a blank string.
Streaming
A stream follows the same walk with one extra rule: it can only change its mind before the first word.
Everything that can refuse happens first, so a page can answer a refusal as normal JSON. Then the gateway walks the chain until the first piece of text arrives. A failure up to that point cost the caller nothing, and the next model can take over. After it, a switch would repeat text that's already on the screen, so the stream ends with one error event instead.
A role that needs guaranteed JSON can't stream at all. The guarantee needs a retry after the answer is complete, and by then a stream has already delivered it.
The ledger
Every call writes one row: tokens, pictures, cost in dollars, the role, the model, the key and a request id. No prompt text goes in it, so nobody's words end up in a cost table.
Each attempt is priced at its own model's rate, so a fallback doesn't get billed at the first model's price. An attempt that never came back isn't billed at all. The row is sent before the gateway returns and before it throws, because a JSON retry spends real tokens even when the call then fails.
Saving the row can fail too. It's retried twice, and the request id means a lost answer can't bill twice. After that the row waits in a queue, and an hourly job writes it. I don't want a database blip to turn into a hole in the bill.
The spending cap
Each project has a monthly allowance. At 80% and at 100% the owners get an email. At 120% AI stops until the month turns.
The check runs before the call. If you check after, you've already paid. The refusal is a 429 with a Retry-After set to the end of the period, so a well-behaved caller knows there's no point in retrying.
Private roles
Some calls read people's mail. Those roles only go to providers that promise not to keep the data, and the rule is in the table, so no caller can forget it. A call also counts as private if it carries a picture or a file.
The two roles that read mail have no fallback list at all. If the agreed hosts are down, the call fails and the mail waits for the next run. A summary that arrives late is fine. A summary sent to a host I never agreed to is not.
When a private call fails, the log records the status and the kind of failure and nothing else. No provider's error text gets in, because that text can repeat what it was sent.
Pictures are different
If a request carries an image, it goes to edit-image. Otherwise it goes to illustrate. The caller never chooses. It's fixed in the code so nobody can point the picture path at a text model.
Each model gets one attempt, with no retries. A timeout on my side can fire after the provider has already billed the picture, so a retry could pay twice. A failure still falls through to the next model.
A call asks for at most four pictures. That limit lives in the gateway, because it's the last place a number someone typed into a form can be caught before it becomes an invoice.
What broke
A rate limit that needed longer. A 429 used to wait 250 milliseconds, like any other hiccup. In September a provider answered 429 twice, 0.7 seconds apart, and a whole day's article run failed. A rate window needs seconds, so a 429 that doesn't say how long to wait now starts at 2 seconds and doubles.
A model that ignored the count. Asked for two pictures, one model drew one. Until 25 October the gateway returned that one picture and logged a warning, and the caller got less than it asked for. Now a model known to do this gets single requests in parallel, and any other model that comes up short gets asked again, up to what's missing.
An answer with no picture. A model can answer in words, for example when a safety filter trips. That call is still billed. If the gateway returned an empty list, it would look like success everywhere. So no picture is an error.
A log line that said nothing. On 3 October a failed picture logged "not a chat completion" and nothing else. Providers can send a 200 with an error inside it. Now the provider's own message and code reach my error, trimmed, and never the payload, because the payload can echo the prompt.
Pictures that cost zero. For a while the picture calls wrote a cost of 0 in the ledger. It didn't matter while nothing read that number. It did matter once a capped service relied on it. Then it recorded the picture's price and counted the input as nothing, which was fine for a short prompt and wrong as soon as a photo was the input. Now it's the price per picture actually returned, plus what the prompt really cost.
Mine, LiteLLM or OpenRouter?
People ask how this compares with LiteLLM and OpenRouter. The honest answer is that mine isn't competing with either of them. It sits in front of OpenRouter.
OpenRouter is a hosted service. You get one key and one endpoint that speaks the OpenAI format, and hundreds of models behind it. You can send a list of models. If the first one fails, it tries the next, and if one host of a model is down, another host can answer. You pay for the model that answered, and you run nothing yourself.
LiteLLM is open source and you run it yourself, either as a Python library or as a proxy server. Its docs say it talks to more than 100 LLMs. The proxy has its own keys, budgets per key, user and team, spend tracking, retries, fallbacks, load balancing, caching and logging. It's the closest thing to what I built, and it's much bigger.
Mine is a thin layer that sends every call on to OpenRouter. What it adds are the rules that only make sense next to my own data.
| OpenRouter | LiteLLM | Mine | |
|---|---|---|---|
| Where it runs | On their servers | On yours | Inside my back office |
| The caller names | A model, or a list of models | A model name you define | A role |
| When a model fails | The next model in your list, or another host | Retries, then the next model; separate lists for some errors | Two tries, then the next model, sorted into three groups |
| Spending limit | A credit limit per key, with an optional reset | Budgets per key, user, team or customer | Monthly per project, checked before the call |
| Private calls | Can be limited to hosts that keep no data | Up to the providers you set up | Hosts that keep no data, and no fallback past them |
| Who keeps it up | They do | You do | I do |
When I'd pick each
Pick OpenRouter when you want many models behind one key and nothing to run. That's why mine uses it.
Pick LiteLLM when several teams share models and each needs its own keys and budget, or when you call many providers directly and want one format for all of them. You run it yourself, and its budgets read spend from a database, so you run that too.
Write a small layer of your own when the rules that matter are tied to your own data: which project pays, what may leave the building, and how long a page may wait.
What mine does that a hosted router doesn't
Callers name a role, never a model. Changing a default is one line and no app changes.
The cap is checked against the same table the invoices read, per project, before the call. When it says no, the refusal tells the caller when the month ends.
The two roles that read mail have no fallback list at all. OpenRouter can keep a call on hosts that keep no data. But the rule that a mail summary waits rather than going somewhere else is mine, and it lives in the table.
And the picture rules from above: at most four, no retries, and an answer without a picture counts as an error.
What they do that mine doesn't
OpenRouter moves a call to another host of the same model, can route by price, speed or latency, and can pick the model for you. I'd never build any of that.
LiteLLM has caching, guardrails, team budgets and a direct line to each provider, without a middleman.
Both are used by far more people than mine, so someone else finds their bugs first.
A hosted router also has a cost of its own. Every prompt passes through one more company, so I read OpenRouter's data settings the way I'd read a contract.
Should you build one?
If you have one app and one provider, no. Your provider's SDK retries well enough.
Once you have several apps, or want to switch models without a release, you want this. I'd look at the open source gateways first: LiteLLM, Bifrost and TensorZero all do far more than mine. Mine lives inside the same back office as the mail and the billing, and the part I needed was small.
If you do build it, go in this order: the table and the required time limit, then the three failure groups, then the ledger, then the cap, because the cap reads the ledger.
