Model Studio throttles application calls on two dimensions: per application and per account. Requests that exceed the limits are rejected with HTTP 429 and the error code Throttling.RateQuota, and usually recover within one minute.
The two dimensions take effect independently, and a throttling error is returned when either limit is reached.
Workflow asynchronous task submissions share the per-application QPS quota with synchronous calls, so the submission rate is throttled in the same way.
LLM-type nodes in a workflow application, such as the LLM, intent classification, parameter extraction, multimodal generation, and LLM-with-retrieval nodes, call models directly and are subject to the RPM/TPM quota of the model that is called. This constraint is independent of application-level throttling, and a request fails when either limit is reached. For model throttling, see Rate limits.
The HTTP status code is
If you expect a significant increase in concurrency or call volume, contact your Alibaba Cloud account manager in advance to request a quota increase. You can also request an increase through the same channel after throttling occurs.
Throttling rules
| Dimension | How it is counted | Default limit |
|---|---|---|
| Per application | Requests per second, counted by application ID (app_id) | 100 QPS |
| Per account | Requests per minute, counted for the same Alibaba Cloud account. Covers all API requests of all applications under the account. | 15,000 QPM |
Model throttling for workflow applications
LLM-type nodes in a workflow application, such as the LLM, intent classification, parameter extraction, multimodal generation, and LLM-with-retrieval nodes, call models directly and are subject to the RPM/TPM quota of the model that is called. This constraint is independent of application-level throttling, and a request fails when either limit is reached. For model throttling, see Rate limits.
Throttling behavior
The HTTP status code is 429 and the response error code is Throttling.RateQuota. SDK calls throw the corresponding throttling exception. Requests usually recover within one minute. Stop retrying during this period to avoid triggering throttling continuously. For the full list of error codes, see Error codes.
How to avoid throttling
- Retry with exponential backoff: After you catch a
429error, wait before you retry, and double the wait time after each failure, such as 1s, 2s, 4s, and 8s. Set a maximum retry count and a maximum total wait time. - Smooth client-side peaks: Add a request queue or a token bucket on the caller side to spread traffic bursts across the time window and avoid concentrated calls within a short period.
- Distribute across applications: The per-application QPS quota is an independent dimension. Split your business into multiple applications by scenario to share the concurrency load. Note that the account-level limit is still counted across all applications.
- Fall back to a standby application: Maintain a list of standby applications, and switch to a standby application when the primary application is throttled.
- Use asynchronous calls and batching: Route requests that do not require a real-time response to workflow asynchronous tasks or batch processing. You can view the status and results of asynchronous tasks in the Task Center.
- Request a quota increase in advance: If you expect significant business growth, contact your Alibaba Cloud account manager to request an increase before you scale up.
FAQ
What do I do if a workflow reports model throttling?
What do I do if a workflow reports model throttling?
The error code is also
Throttling.RateQuota, but the throttling is triggered by the model that a workflow node calls, and is unrelated to application-level throttling. Check the RPM/TPM quota of the model. For more information, see Rate limits.