HTTP protocol

The HTTP protocol in the multimodal interaction development suite supports sending http-sse requests with plain text or images to retrieve results from the large model.

When you use the HTTP protocol to access the multimodal interaction development suite, the following features are not supported:

Pure vision applications are not supported.
The following agents are not supported: Video Call, Express Video/Voice Call, Voice Translation, News Radio, Children's Stories, Audio Recording Summary, Proactive Tour Guide, and Proactive Companion.

Prerequisites

You must activate the service and obtain and configure an API key. To prevent security risks, configure the API key as an environment variable instead of hard coding it in your code.

For client-side calls, handling the API key on the client poses a security risk. You can obtain a temporary authentication token from the server-side using your API key, and then send the token to the client for use. For more information, see Generate a temporary API key.

Endpoint

https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation

When you send an HTTP request, the header must include X-DashScope-SSE: enable.

Authentication

When you send an HTTP request, include the API key in the HTTP header. Replace your_api_key with your actual API key.

"Authorization": "Bearer your_api_key"

Message types

The HTTP protocol supports only text messages, which are JSON-formatted strings.

Send a request

Request - Input Message

You can send an HTTP request.

Top-level parameter	Second-level parameter	Type	Required	Description
model		string	Yes	The name of the Alibaba Cloud Model Studio model. This is fixed to "multimodal-dialog". Copy and use this value directly.
input	directive	string	Yes	Instruction name: Request
	app_id	string	Yes	The ID of the application created by the customer (Get APP ID). You can view this ID on the "My Applications" page of the multimodal interaction development suite console.
	dialog_id	string	No	The dialog ID. If you leave this empty, a new dialog is started. The server automatically generates a dialog ID and returns it in the response. Example format: "12345678-1234-1234-1234-1234567890ab" (36 characters). To continue a previous conversation, pass the dialog_id that the server returned.
	text	string	Yes	The text to process. When calling some agents, `text` can be an empty string (""). The server then uses the `images` or `biz_params` parameters for processing. For more information, see Call official agents.
parameters	client_info	object	Yes	For parameter descriptions, see the parameters.client_info parameters table below.
	images	list[]	No	The image data to analyze. Only multimodal applications support image Q&A. For parameter descriptions, see the parameters.images parameters table below.
	biz_params	object	No	Configure as needed. For parameter descriptions, see the parameters.biz_params parameters table below.

The following table describes the parameters for parameters.client_info.

Top-level parameter	Second-level parameter	Type	Required	Description
user_id		string	Yes	The end user ID. Generate this ID based on your business rules to implement customized features for different end users. The maximum length is 36 characters.
device	uuid	string	No	A globally unique ID for the client. You must generate this ID and pass it to the software development kit (SDK). The maximum length is 40 characters. An end user can have multiple devices. Each device has a different UUID, but the user_id is the same.
network	ip	string	No	The public IP address of the caller.
location	latitude	string	No	The latitude of the caller. Submit this in scenarios that require the precise location of the client.
	longitude	string	No	The longitude of the caller. Submit this in scenarios that require the precise location of the client.
	city_name	string	No	The city where the caller is located. This indicates the approximate location of the client.

The following table describes the parameters for parameters.images.

Top-level parameter	Type	Required	Description
type	string	Yes	The image type. Supported types are base64 and url.
value	string	Yes	The image content. If `type` is base64, this is the base64 string of the image. If `type` is url, this is the URL of the image.

Top-level parameter

Type

Required

Description

type

string

Yes

The image type. Supported types are base64 and url.

value

string

Yes

The image content.

If `type` is base64, this is the base64 string of the image.
If `type` is url, this is the URL of the image.

The following table describes the parameters for parameters.biz_params.

Top-level parameter	Type	Required	Description
user_defined_params	json object	No	Parameters to pass through to the agent. For information about the parameters passed by different agents, see Call official agents.
user_prompt_params	json object	No	Used to set custom prompt variables. You can define the key and value in the JSON object. For information about how to configure custom prompt variables in the console, see Application configuration - Prompts.
user_query_params	json object	No	Used to set custom dialog variables. You can define the key and value in the JSON object. For information about how to configure custom dialog variables in the console, see Application configuration - Dialog variables.

Example

{
    "model": "multimodal-dialog",
    "input": {
        "directive": "Request",
        "app_id": "****************",
        "text":"Hello, what is the weather like in Beijing today?"
    },
    "parameters": {
        "client_info": {
            "user_id": "bi**********07",
            "device":{
              "uuid": "432k*********k449"
            },
            "network":{
              "ip": "x.x.x.x"
            },
            "location":{
               "city_name": "Beijing"
            }
        },
        "images":[{
              "type": "base64",
              "value": "base64String"
          }]
    }
}

Text event

RespondingContent - Output Message

Top-level parameter	Second-level parameter	Type	Required	Description
output	event	string	Yes	Event name: RespondingContent
	dialog_id	string	Yes	Dialog ID
	round_id	string	Yes	The ID of the current interaction round.
	llm_request_id	string	Yes	The request_id for the large language model (LLM) call.
	text	string	Yes	The text output from the system. This is a streamed, incremental output.
	spoken	string	Yes	The text used for speech synthesis. This is a streamed, incremental output.
	finished	bool	Yes	Indicates whether the output is complete.
	finish_reason	string	No	The reason the output finished. Currently, only one reason is available: stop indicates a normal finish.
	extra_info	object	No	Other extension information. The following are supported: commands: A command string. This field is a JSON string and requires secondary parsing. For information about the command strings used by different agents, see Call official agents. agent_info: Agent information. tool_calls: Information returned by the plugin. tool_infos: Tool information.

Example

{
    "output": {
        "event": "RespondingContent",
        "dialog_id": "e788xxxx-xxxx-xxxx-xxxx-xxxx9637",
        "round_id": "d8****************************29",
        "llm_request_id": "fb****************************67",
        "text": "Hello",
        "spoken": "Hello",
        "finished": true,
        "finish_reason":"stop",
        "extra_info": {
            "agent_info": {
                "round": 1,
                "device": {
                    "device_id": ""
                },
                "intent_infos": [
                    {
                        "intent": "***",
                        "domain": "***"
                    }
                ]
            },
            "query": "Hello, what is the weather like in Beijing today?"
        }
    },
    "request_id": "6481xxxx-xxxx-xxxx-xxxx-xxxx1aed0"
}

Error event

This event provides error information. For a description of error codes, see Multimodal interaction suite - Error codes.

Error - Output Message

Top-level parameter	Type	Required	Description
code	string	Yes	Error code
message	string	Yes	Error message
request_id	string	Yes	Request ID

{
    "code": "InternalLLMError",
    "message": "Internal LLM error",
    "request_id": "0aa0xxxx-xxxx-xxxx-xxxx-xxxxaa21"
}

Call examples

The following is an example of a request. The header must include X-DashScope-SSE: enable.

Replace your_api_key, app_id, text, and other parameters in the parameters object with your actual values.

curl

curl --location "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation" \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer your_api_key" \
--header "X-DashScope-SSE: enable" \
--data '{
    "model": "multimodal-dialog",
    "input": {
            "directive": "Request",
            "app_id": "****************",
            "text":"Hello, what is the weather like in Beijing today?"
        },
    "parameters": {
            "images":[{
              "type": "base64",
              "value": "base64String"
                }],
            "client_info": {
                "user_id": "bin********207",
                "device":{
                  "uuid": "432k*********k449"
                },
                "network":{
                  "ip": "x.x.x.x"
                },
                "location":{
                  "city_name": "Beijing"
                }
            }
        }
}'

​Prerequisites

​Endpoint

​Authentication

​Message types

​Send a request

​Request - Input Message

Example

​Text event

​RespondingContent - Output Message

Example

​Error event

​Error - Output Message

​Call examples

Prerequisites

Endpoint

Authentication

Message types

Send a request

Request - Input Message

Text event

RespondingContent - Output Message

Error event

Error - Output Message

Call examples