Deploy a trained text model as an online API service.
Prerequisites
- Supported region: The model deployment API is currently only available in the China (Beijing) region. If you are using a different region, please complete the model deployment through the Model Studio console for that region.
- Account permissions: If using an Alibaba Cloud sub-account (RAM user), you need to grant the sub-account permissions for model invocation, training, and deployment.
- Configure environment variables: You have successfully obtained an API Key and configured it as an environment variable.
- Read deployment documentation: It is recommended to first read Dedicated deployment overview and Create a model deployment using API to understand the usage and basic steps of model deployment.
Create a deployment
- China (Beijing)
POST https://dashscope.aliyuncs.com/api/v1/deploymentsFor Windows CMD, replace$DASHSCOPE_API_KEYwith%DASHSCOPE_API_KEY%. For PowerShell, replace it with$env:DASHSCOPE_API_KEY
Request parametersHeadersContent-Typestring (Required)Fixed value: application/jsonAuthorization string (Required)API Key authentication, in the format Bearer sk-xxxx.Request Bodymodel_namestring (Required)The name of the model to deploy, corresponding to the Model ID in My Models. You can also obtain it from the output of the Create a tuning job API.plan string (Required)Deployment plan. Valid values:
string (Conditionally required)Deployment template. Required when plan is mu. Example: "deploy_spec": "MU1".You can obtain it from the template_id field returned by the List deployable models API.capacity integer (Conditionally required)The number of resource units for the deployment, which must be an integer multiple of base_capacity. Different deploy_spec values have different constraints. For example, MU2 must be a multiple of 8, while MU5 can be 1.billing_method string (Conditionally required)Billing method. Required when plan is mu. Currently supports "POST_PAY" (pay-as-you-go).enable_thinking boolean (Optional)Only configurable when plan is mu. Supported by some models. Can be set to true or false.max_context_length number (Optional)Only configurable when plan is mu. Supported by some models. Example: "max_context_length": 131072.rpm_limit number (Optional)Only configurable when plan is mu. Supported by some models. Requests per minute.tpm_limit number (Optional)Only configurable when plan is mu. Supported by some models. Tokens per minute.ptu_capacity object (Optional)Only takes effect when plan is ptu. If not provided, defaults to 10,000 input_tpm and 1,000 output_tpm.
ptu_capacity properties input_tpm numberThe maximum input tokens per minute supported by the deployed model.output_tpm numberThe maximum output tokens per minute supported by the deployed model.thinking_output_tpm numberSupported by some models. The maximum thinking output tokens per minute supported by the deployed model.string (Optional)The display name of the model in the console. If not provided, the value of model_name will be used as the deployment name.suffix string (Optional)After model deployment, a new model name will be generated. The suffix is used to specify the suffix of the new model name, with a maximum length of 8 characters and must be globally unique. When deploying a model for the first time, the suffix can be omitted. For subsequent deployments of the same model, a suffix must be specified to distinguish between them. |
Billing is based on the usage duration of model units. Suitable for large-scale inference after model fine-tuning, with dedicated resources and flexible performance and cost control. After executing the deployment command below, billing will start once the deployment is successful, even if you have not yet invoked the model. Please confirm the billing rules before executing the deployment command. |
Deployment troubleshooting and performance tuning
max-num-seqs parameter is not configurable
Under both provisioned throughput billing and model unit billing plans, throughput, concurrency, and generation speed are all preset by the platform. Direct adjustment of inference engine parameters such as max-num-seqs in vLLM is not supported, and this parameter does not exist in the request parameters of this API.
Under the model unit billing plan, you can indirectly control throughput by selecting the model unit type (corresponding to the request parameter deploy_spec) and adjusting the number of deployment replicas (corresponding to the request parameter capacity).
Deployment template and resource isolation
Currently, the deployment template only supports single-machine deployment (Single-machine deployment - Enhanced general inference) and does not support multi-GPU instance isolation. For resource isolation, use the model unit billing plan, which provides dedicated computing resources for your workload.
Throttling error handling
When concurrent requests exceed the throttling threshold, the API returns HTTP 429 with error code Throttling.RateQuota and error message Requests rate limit exceeded, please try again later. You can handle this as follows:
- Provisioned throughput billing: Adjust
ptu_capacityvalues forinput_tpmandoutput_tpm(corresponding to Input kTPM and Output kTPM in the console), or reduce the request frequency. Under this plan, the overflow strategy can be set to Auto overflow (switch to pay-as-you-go billing) or Use PTU capacity only (requests exceeding capacity will directly return429). - Model unit billing: Adjust
rpm_limitandtpm_limit, or reduce the request frequency.
Context length tuning
Under the model unit billing plan, you can configure the maximum context length via the request parameter max_context_length, with a value range of 1 to 262144 (the actual upper limit depends on the capabilities of the deployed model). This parameter limits the context size per request, thereby limiting the memory usage per request and reducing the risk of OOM errors.
When processing tasks with a large number of images, set this parameter based on the image dimensions and the number of images per request.
Response parametersrequest_idstringThe unique identifier of the request.output objectTask details.
Properties deployed_model stringThe unique identifier of the deployed model. Used to query deployment status and invoke the model.model_name stringThe model identifier name.status stringDeployment status:
stringThe base model used.gmt_create stringThe creation time of the deployment task.gmt_modified stringThe last modification time of the deployment task.workspace_id stringThe workspace ID associated with the Alibaba Cloud Model Studio API Key. See Get the Workspace ID.charge_type stringBilling mode. post_paid indicates pay-as-you-go.creator stringThe Alibaba Cloud account ID of the creator.modifier stringThe Alibaba Cloud account ID of the last modifier.plan stringThe deployment plan.base_capacity numberThe minimum number of resource units required to run the base model.ready_capacity numberThe number of resource units that are ready and can immediately process requests.model_unit_spec stringThe model unit specification. Only returned when plan is mu.enable_thinking booleanWhether thinking mode is enabled. Only returned when plan is mu.max_context_length numberThe maximum context length limit. Only returned when plan is mu.rpm_limit numberRequests per minute. Only returned when plan is mu.tpm_limit numberTokens per minute. Only returned when plan is mu.ptu_capacity objectProvisioned throughput configuration. Only returned when plan is ptu.
ptu_capacity properties input_tpm numberThe maximum input tokens per minute supported by the deployed model.output_tpm numberThe maximum output tokens per minute supported by the deployed model.thinking_output_tpm numberSupported by some models. The maximum thinking output tokens per minute supported by the deployed model.stringError code. Returned when the call fails.message stringDetailed error description. Returned when the call fails. |
Key fields: output.deployed_model (unique identifier of the deployed model) and output.status (deployment status). |