Sources checked
Provide a route for uncertain requests
For an original help-desk design, route known account questions to a retrieval workflow and unfamiliar technical failures to an investigation queue. Keep a fallback path for uncertain classification rather than forcing every request into a confident label.
Routing errors can erase the savings
A cheaper destination can still be expensive if incorrect routing creates retries or manual repair. Measure routing mistakes and end-to-end outcomes together.
What a route is allowed to depend on
A route can look at the task type, the input length, a user tier, or a classifier’s label. Those are inspectable.
A route that says “use the best model” is not a rule. Name the model id for each branch.
Include the fallback when the preferred model errors or is over its rate limit. A fallback that nobody tested is a second product.
The classifier that picks the route is itself a model or a heuristic. It can be wrong in a systematic way.
A long legal question mislabeled as chitchat will be answered by the small model. Measure that confusion on your own traffic sample.
Do not tune the router on the playground examples that were written to be obvious.
What to log so the route can be undone
Store the router input, the branch, the model id, and the reason. Without the reason, a bad answer cannot be traced to a policy.
Also store the cost. A router that saves money on average can still send the expensive cases to the expensive model and add the router’s own latency to every request.
When you change a model id, the route changes even if the rule text stays. Pin versions.
Re-run the boundary cases: the shortest input that should escalate, and the longest input that should stay on the small model. Those two rows catch more regressions than a chart of average quality.
What to record before you rely on it
Write the routing table as data: condition, model id, fallback, and the date it was approved. Keep twenty boundary examples with the branch a person expected.
Alert if a branch’s share of traffic moves suddenly, because a classifier drift will not show up as an outage.
Keep the model id, the date, and the inputs you used beside that note, so a later change can be compared with the same test. When the result is a failure, store the artifact and change the design.
Do not relax the check until the model happens to pass. Name a person who owns that failure.
A note with no owner and no inputs cannot be repeated, and a pass with no date cannot be compared with the next model id.
Common questions
Should every request try the cheap model first?
Only if a failed cheap attempt is cheap to detect and safe to retry. If the cheap model can take an action, a “try then escalate” policy may already have acted.
Is a router a way to avoid evaluating models?
No. Each branch needs its own eval. The router adds a further eval: whether the right branch was chosen.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards





