Smart routing
Smart routing distributes requests across service providers that offer the same model. Your application keeps using the model ID in its request; routing selects a service provider and never substitutes a different model.
How a route is selected
The gateway first applies authentication, budget, rate-limit, model-access, protocol, modality, and health checks. It then scores the remaining service providers using price, time to first token (TTFT), total latency, throughput, and failure risk. Recent request telemetry continuously updates the selection probabilities.
The modality check reads the request content. Requests that carry image, audio, or video input are routed only to service providers that declare the matching understanding capability for that model. If no available provider declares it, the gateway returns INPUT_MODALITY_NOT_SUPPORTED instead of forwarding the request to a provider that cannot process the input.
Routing modes
| Mode | Primary emphasis | Best suited to | Trade-off |
|---|---|---|---|
| Balanced | Price, TTFT, latency, throughput, and failure risk | Most online workloads and new projects | Does not optimize a single metric to the extreme |
| Price first | Gives price the largest share of the combined score | Batch and offline generation | Latency may be higher |
| Speed first | Gives TTFT, latency, and throughput the largest combined share | Chat, code completion, and other interactive requests | May select a higher-priced service provider |
Balanced is the default starting point. InOneAPI currently supports these three modes only.
Price first
Price-first routing tends to send more traffic to lower-priced service providers for the same model. It does not replace the requested model with a cheaper one, and it does not blindly pin every request to the lowest price. Health, failure risk, and exploration traffic still influence the selection.
Speed first
Speed-first routing emphasizes recent TTFT, end-to-end latency, and throughput. Failure risk and price remain part of the score, so speed first does not mean ignoring reliability or cost.
Balanced
Balanced routing gives all five signals meaningful weight and is the default for new projects. Health checks, abnormal-traffic detection, and circuit breaking apply to every mode; there is no separate availability-first mode.
Pin a service provider
Set the optional X-InOneAPI-Provider request header to the provider name shown on the model detail page, for example X-InOneAPI-Provider: Tencent Cloud. This pins the request to that provider and bypasses price, speed, and balanced provider selection. The gateway may rotate or retry among that provider's configured API keys, but it never fails over to a different provider. If the provider does not offer the requested model and API, the gateway returns PROVIDER_NOT_AVAILABLE. Temporary provider unavailability also fails the request without cross-provider fallback. Copy the provider name from the model detail page to use it.
Verify a mode
Observe at least one representative workload cycle and compare success rate, P50/P95 latency, TTFT, retry rate, and cost per request. Use request records and X-Gateway-Trace-ID to distinguish routing changes from service-provider or traffic changes.