Smart model routing that keeps governance intact
Cost-aware routing can match each workload to an approved model that fits its cost, quality, and policy needs. Here is how to pursue that savings potential and keep control.
By Halo Product Team
Many requests do not need the most capable model on the menu. A short summary, a reformatted table, or a simple classification can often run on a smaller and cheaper model. Smart model routing matches each workload to an approved model that fits its cost, quality, and policy needs, and that match is where the savings potential comes from.
Routing also means a system makes decisions on your behalf. If it picks a model your policy has not approved, or retries on a provider your data rules exclude, the savings arrive with a governance problem. Governance has to be part of the route from the start.
Start with an approved model list
Routing choices come from a list, so build the list first. For each model, record the provider, the price, the data classes it is approved for, and the workloads that may use it.
A model catalog with pricing gives the router its cost input. Approval status gives it a boundary. A model that is cheap and unapproved should never appear as a candidate.
Cost is one input to a route. Approval, data rules, and quality needs set the boundary around it.
Run policy before routing
Order matters. Evaluate policy first, then route. The sequence below shows one way to do it.
- Identify the caller. The request carries an agent identity, so the right rules apply.
- Check policy. Rules evaluate the request and can allow it, deny it, or raise an alert. Selected classifiers can also mask sensitive values in supported gateway paths.
- Build the eligible set. Keep only the models approved for that agent and that kind of data.
- Pick by profile. Within the eligible set, a route profile chooses by cost, quality, or latency, depending on the workload.
- Fall back inside the set. If the chosen model is unavailable, try the next approved model.
- Record the decision. Keep what policy decided, which model served the request, and what it cost.
With policy first, the router only ever chooses among models that the request was already allowed to use.
Give fallbacks the same scrutiny
Providers have outages and rate limits, so a route needs somewhere to go. A fallback is a second routing decision, and it deserves the same review as the first.
- Stay inside the approved set. A fallback model needs approval like any other.
- Match the data rules. The fallback should be approved for the same data classes as the primary model.
- Record the switch. A reviewer should be able to see that a request used a fallback, and why.
- Decide what happens when nothing is available. A clear failure beats an unapproved model.
The post on what happens when an AI control goes offline covers failure behavior in more depth.
Route profiles for real workloads
A route profile describes what matters for a class of work. For example, a team might define three profiles.
| Profile | Priority | Example workloads |
|---|---|---|
| Economy | Lowest cost among approved models | Summaries of internal notes, formatting, tagging |
| Balanced | Good quality at a reasonable cost | Draft replies, code explanations |
| Careful | Highest quality among models approved for sensitive data | Contract review, financial analysis |
These profiles are hypothetical. Your own workloads, data classes, and quality bar decide how many profiles you need and what each one allows.
Picture a support agent that drafts replies under the Balanced profile. Most drafts go to a mid-range approved model.
If that model is rate limited, the gateway tries the next approved model on the fallback path. If a policy check flags a sensitive value, the gateway can mask it in supported paths. The decision record shows each step.
Know when to pin a model
Some workloads should stay on one approved model. A reviewer may have validated outputs for a specific model, or a regulated process may name it.
In those cases, pin the workload to that model and exclude it from cost-based routing. Pinning keeps the approval and the evidence tied to the same thing.
Mistakes to avoid
Four mistakes are easy to make in a routing plan.
- Routing on price alone. A cheap model that needs many retries or manual fixes can cost more per accepted result.
- Leaving fallbacks stale. When the approved list changes, review each fallback path too.
- Skipping reruns. A provider can change a model's behavior over time, so rerun your task set on a schedule.
- Treating estimates as invoices. Reconcile cost analytics against provider bills, and note the gaps.
Measure the savings potential honestly
Savings potential is a hypothesis until you test it on your own work. Before you move a workload, pick a fixed set of representative tasks, run them through the current model and the candidate, and compare quality and cost side by side.
Use cost analytics by agent and model to find where spend concentrates, since routing matters most there. Your prompts, volumes, and quality bar determine the result, so measure on your own workloads before you commit to a number.
Compare cost against accepted work as well as token counts, and label what your cost view cannot see. The posts on connecting AI spend to AI value and coverage labels for cost estimates cover both points.
Add budgets as a backstop
Routing improves unit cost, and budgets limit the downside. Set account and per-agent budgets with thresholds, so a loop or a misrouted workload triggers a notification or a pause before it becomes a surprise. The post on building budgets that follow the owner explains how to choose thresholds and actions.
How Halo helps
Halo's model gateway supports configurable route profiles, model catalogs with pricing, and fallback paths, and it applies identity, policy, budgets, and evidence to agent traffic. Model access can be Halo-managed or BYOK, with BYOK providers including OpenAI, Anthropic, and Google Gemini, and Halo-managed models run through Amazon Bedrock. Read about model routing or book a walkthrough to review one workload.