Smart routing
Smart routing distributes requests across service providers that offer the same model. Your application keeps using the model ID in its request; routing selects a service provider and never substitutes a different model.
How a route is selected
The gateway first applies authentication, budget, rate-limit, model-access, protocol, modality, and health checks. It then scores the remaining service providers using price, time to first token (TTFT), total latency, throughput, and failure risk. Recent request telemetry continuously updates the selection probabilities.
Routing modes
| Mode | Primary emphasis | Best suited to | Trade-off |
|---|---|---|---|
| Balanced | Price, TTFT, latency, throughput, and failure risk | Most online workloads and new projects | Does not optimize a single metric to the extreme |
| Price first | Gives price the largest share of the combined score | Batch and offline generation | Latency may be higher |
| Speed first | Gives TTFT, latency, and throughput the largest combined share | Chat, code completion, and other interactive requests | May select a higher-priced service provider |
Balanced is the default starting point. InOneAPI currently supports these three modes only.
Price first
Price-first routing tends to send more traffic to lower-priced service providers for the same model. It does not replace the requested model with a cheaper one, and it does not blindly pin every request to the lowest price. Health, failure risk, and exploration traffic still influence the selection.
Speed first
Speed-first routing emphasizes recent TTFT, end-to-end latency, and throughput. Failure risk and price remain part of the score, so speed first does not mean ignoring reliability or cost.
Balanced
Balanced routing gives all five signals meaningful weight and is the default for new projects. Health checks, abnormal-traffic detection, and circuit breaking apply to every mode; there is no separate availability-first mode.
Verify a mode
Observe at least one representative workload cycle and compare success rate, P50/P95 latency, TTFT, retry rate, and cost per request. Use request records and X-Gateway-Trace-ID to distinguish routing changes from service-provider or traffic changes.