Skip to main content

Custom instruction practices

This document describes standards for configuring multimodal instructions and enabling device interaction. It covers instruction design, time formats, interaction flows, and developer support. The goal is to seamlessly connect intent models with device control, establishing a standardized and reliable interaction process.

Introduction to instructions

The multimodal interaction cloud sends instructions to terminals to translate user intents into specific actions. Use instructions to extend features and control device hardware, media playback, and applications. Examples include turning devices on or off, adjusting settings, managing playback progress, or navigating within an application. Instructions are highly extensible. A set of common preset instructions is provided, and you can also create custom instructions as needed. For more information about how to configure instructions in the console, see Application configuration.

Instruction configuration tips

To ensure the intent model accurately understands and invokes the correct tool, follow strict tool definition standards. The following key points and examples help you define tools effectively.

Write clear and accurate tool descriptions

  • Goal: Help the intent model understand what the tool does.
  • Note: Avoid vague language. Ensure the description matches the tool’s actual function.
Example:
  • Good: "Query the weather for a specific date"
  • Bad: "Check weather" (Too broad. It might be mistaken for other weather-related operations.)

Define parameters without ambiguity

  • Goal: Clearly define the purpose and value range for each parameter.
  • Recommendation: Use enumerations to limit parameter options whenever possible. Enumeration definition is not yet available, so define them in the description. For open-ended inputs, provide a detailed description.
Example — Define a tool to play music: 
{
  "type": "object",
  "required": ["action", "media"],
  "properties": {
    "action": {
      "type": "string",
      "description": "Playback control action",
      "enum": ["play", "pause", "stop"]
    },
    "media": {
      "type": "string",
      "description": "Media file name or URL"
    }
  }
}

Be concise but comprehensive

  • Goal: Keep tool definitions simple but complete.
  • Note: Do not list too many examples. Include only necessary explanations.
Example — Tool definition for a query: 
{
  "name": "query_city_news",
  "description": "Query news for a specific city",
  "parameters": {
    "type": "object",
    "required": ["city"],
    "properties": {
      "city": {
        "type": "string",
        "description": "The name of the city to query"
      },
      "date": {
        "type": "string",
        "description": "The date to query, in yyyy-mm-dd format. If empty, queries the current date."
      }
    }
  }
}

Merge similar parameters logically

When designing tool calling interfaces, merging similar parameters means combining multiple parameters that have similar functions or can be processed together into a single parameter. This simplifies the interface definition, reduces redundancy, and makes tool calls more intuitive. The following examples show how to merge parameters logically:

Example 1: Control a media player

Suppose you have two separate tool functions to control a media player’s state: play_media to start playback and pause_media to pause it. Both tools control the playback state. You can merge them into a single tool, control_playback, and use an action parameter to specify the command.
Original definition
  • play_media
    • Parameters: None
  • pause_media
    • Parameters: None
Merged definition
  • control_playback
    • Parameters:
      • action: {"type": "string", "description": "Specifies the action to perform (for example, 'play' or 'pause')", "enum": ["play", "pause"]}
This approach reduces the number of tools to maintain and makes playback state control more flexible and consistent.

Example 2: Set a reminder

Consider an application scenario for setting alarms with two different tools: set_alarm for a one-time alarm and set_repeating_alarm for a recurring alarm. Both tools configure alarms but have different trigger conditions. You can merge them by adding a new parameter for the repeat pattern.
Original definition
  • set_alarm
    • Parameters:
      • time: Human-readable time
  • set_repeating_alarm
    • Parameters:
      • repeat_pattern: Human-readable recurring date
      • time: Human-readable time
Merged definition
  • set_alarm
    • Parameters:
      • time: Human-readable time
      • repeat_pattern: {"type": ["string", "null"], "description": "If set, this is a recurring alarm. Otherwise, it is a one-time alarm. Supports human-readable recurring date formats. The default value is null, which indicates a non-recurring alarm."}
In this case, adding the optional repeat_pattern parameter lets you support both one-time and recurring alarms. This avoids creating two very similar tools.

Summary

To merge similar parameters logically, identify which parameters perform the same or highly related tasks. Then, find a way to unify these behaviors. This simplifies system design, improves code maintainability, and enhances user experience. However, do not overcomplicate a single tool’s function when merging parameters. This can cause confusion or difficulty in use.

Human-readable time format 

{
        "humanReadableDate": {
            "title": "Human-readable date",
            "type": "string",
            "x-type": "humanReadableTime",
            "description": "Can be an absolute, relative, or standard date. The system automatically parses it into yyyy-mm-dd format.",
            "examples": [
                "Tomorrow",
                "Next Tuesday",
                "Three days later"
            ]
        },
        "humanReadableTime": {
            "title": "Human-readable time",
            "type": "string",
            "x-type": "humanReadableTime",
            "description": "Can be an absolute or relative time described in natural language. The system automatically parses it into HH:MM:SS format.",
            "examples": [
                "In three hours",
                "5 PM today",
                "12:15"
            ]
        }
    }
If a parameter name starts with date_, the system interprets it as a humanReadableDate. If a parameter name starts with time_, the system interprets it as a humanReadableTime.

Custom tool JSON file format

You can directly upload definitions for all tools. The file must be in JSON format. The JSON schema is as follows.

1. Definition structure 

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "LLM Tool Definition with Examples",
  "type": "object",
  "required": ["tool_definition", "usage_examples"],
  "properties": {
    "tool_definition": {
      "type": "object",
      "description": "Tool function definition that complies with Large Language Model (LLM) standards",
      "required": ["name", "description", "parameters"],
      "properties": {
        "name": {
          "type": "string",
          "description": "The unique identifier for the tool. We recommend using snake_case."
        },
        "description": {
          "type": "string",
          "description": "A detailed description of the tool's function, so the model can determine when to call it."
        },
        "parameters": {
          "type": "object",
          "description": "A subset of the JSON Schema for parameter definitions."
        }
      }
    },
    "response_type": {
      "type": "string",
      "description": "The response type after the tool is called.",
      "enum": ["ON_EXECUTION_RESULT", "IMMEDIATE"],
      "default": "ON_EXECUTION_RESULT"
    },
    "usage_examples": {
      "type": "array",
      "description": "An array of few-shot examples for the model to reference.",
      "items": {
        "type": "object",
        "required": ["text", "tool_call"],
        "properties": {
          "text": {
            "type": "string",
            "description": "The user's natural language query text."
          },
          "tool_call": {
            "type": "object",
            "description": "The parameter result that should be generated for the preceding text.",
            "required": ["arguments"],
            "properties": {
              "arguments": {
                "type": "object",
                "description": "Specific key-value pairs for the parameters."
              }
            }
          }
        }
      }
    }
  }
}

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "LLM Tool Set Definition",
  "type": "object",
  "required": ["set_name", "tools"],
  "properties": {
    "set_name": {
      "type": "string",
      "description": "The logical name of the tool set, such as 'office_automation' or 'social_media_tools'"
    },
    "description": {
      "type": "string",
      "description": "An overall description of the purpose of this tool set."
    },
    "tools": {
      "type": "array",
      "description": "A list of all tools included in this set.",
      "items": {$JSON schema for a single tool}
    }
  }
}

{
  "tool_set_name": "content_manager",
  "tools": [
    {
      "tool_definition": {
        "name": "create_note",
        "description": "Create a new text note",
        "parameters": {
          "type": "object",
          "properties": {
            "title": { "type": "string", "description": "Note title" },
            "content": { "type": "string", "description": "Detailed content of the note" }
          },
          "required": ["title", "content"]
        }
      },
      "usage_examples": [
        {
          "text": "Help me remember, there's a weekly meeting at 3 PM tomorrow.",
          "tool_call": {
            "arguments": {
              "title": "Schedule Reminder",
              "content": "There's a weekly meeting at 3 PM tomorrow."
            }
          }
        }
      ],
      "response_type": "ON_EXECUTION_RESULT"
    },
    {
      "tool_definition": {
        "name": "add_to_list",
        "description": "Add an item to a specified list",
        "parameters": {
          "type": "object",
          "properties": {
            "list_name": { "type": "string", "description": "The name of the list, such as: shopping list" },
            "item": { "type": "string", "description": "The name of the item to add" }
          },
          "required": ["list_name", "item"]
        }
      },
      "usage_examples": [
        {
          "text": "Add apples to my shopping list",
          "tool_call": {
            "arguments": {
              "list_name": "shopping list",
              "item": "apples"
            }
          }
        }
      ],
      "response_type": "ON_EXECUTION_RESULT"
    }
  ]
}
A complete tool definition includes the following three core parts:

Field name

Type

Required

Description

name

string

Yes

The unique identifier for the tool. It can only contain letters, numbers, and underscores. We recommend using snake_case.

description

string

Yes

Crucial. Describes the tool's function and applicable scenarios. The model uses this description to decide whether to call the tool.

parameters

object

Yes

Defines the parameters the tool accepts. It must be an object that conforms to the JSON Schema.

parametersinternal structure

  • type: Must be "object".
  • properties: Defines the specific key-value pairs. Each key represents a parameter.
  • required: A string array that lists the names of all required parameters.
Meta-definition for the tool list 
{
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string", "pattern": "^[a-zA-Z0-9_-]+$" },
          "description": { "type": "string", "minLength": 1 },
          "parameters": { "type": "object" }
        },
        "required": ["name", "description"],
        "additionalProperties": false
      }
}

Provide model responses when sending instructions

Console configuration

To provide a model response while sending an instruction, select a reply mode on the Settings page:
  • Reply based on execution status (default): After sending the instruction, the system waits for a success or failure message from the device. The model then generates a corresponding reply, such as "The air conditioner has been set to 26 degrees" or "The volume is already at maximum and cannot be increased further."
  • Auto-reply: Automatically outputs a reply when the instruction is sent.
Screenshot

Device-side instruction acceptance

For example, if the unmute skill is configured to require a reply. The result of sending the "unmute" voice command is: 
{
    "payload": {
        "output": {
            "event": "RespondingContent",
            "text": "",
            "spoken": "",
            "finished": true,
            "finish_reason": "stop",
            "extra_info": {
                "tool_infos": [

                ],
                "agent_info": {
                    "intent_infos": [
                        {
                            "domain": "general_command",
                            "intent": "unmute"
                        }
                    ],
                    "round": 1
                },
                "commands": "[{\"intent_info\":{\"domain\":\"general_command\",\"intent\":\"unmute\"},\"command_request_id\":\"35b635f3-6511-450e-8fa1-6955d5279367\",\"name\":\"unmute\",\"params\":[]}]",
                "query": "unmute"
            },
            "dialog_id": "56fe9300-d80c-471a-80bd-20c0296acd93",
            "round_id": "70f636e1d0284bc8a8bac2767e61b4b5",
            "llm_request_id": "1d8f3469274e4d13bbc3b28460eabb76"
        }
    }
}
In the result, payload.output.extra_info.commands is the instruction execution result. When you select "Reply based on execution status", the result includes a command_request_id. Use this ID as the key when replying with the result.

Device-side instruction reply

After you receive an instruction or an Agent result, execute it on the device. You obtain a success or failure result. Sync this result with the server-side by submitting it through the following interface:

Parameter

Level 1 parameter

Level 2 parameter

Level 3 parameter

Level 4 parameter

Description

Skill response/Third-party Agent callback execution result

biz_params

command_results[]

command_request_id

string

Request ID

invoke_result

Execution result

content

Object

Varies by result. For more information, see the unified device-side return specification.

structuredContent

JSON object

Structured return information

structuredContent.success

Boolean

Indicates whether the operation was successful.

invoke_result.content is the data part passed from the device to the Agent. We recommend using the following unified device-side return format:

Parameter name

Type

Meaning

content

Object

Varies by result. For more information, see the text content, image content, and audio content sections below.

Text Content 
{
  "type": "text",
  "text": "Tool result text"
}
Image Content 
{
  "type": "image",
  "data": "base64-encoded-data",
  "mimeType": "image/png"
  "annotations": {
    "audience": ["user"],
    "priority": 0.9
  }
}
Audio Content 
{
  "type": "audio",
  "data": "base64-encoded-audio-data",
  "mimeType": "audio/wav"
}
Example result: 
[
    {
        "command_request_id": "xxxx",
        "invoke_result": {
            "content": {
                "type": "text",
                "text": "Successfully opened"
            },
            "structuredContent": {
                "success": true
            }
        }
    }
]

Request code example - Device-side callback for execution result (Java) 

public static HashMap<String,Object> getAgentOrCommandResult() {

  HashMap<String,Object> commandResult = new HashMap<>();
  commandResult.put("command_request_id", "35b635f3-6511-450e-8fa1-6955d5279367");
  commandResult.put("invoke_result", new Object<>());
    // invoke_result.structuredContent.success = true/false execution result

  ArrayList<Object> commandResults = new ArrayList<>();
  commandResults.add(commandResult);

  HashMap<String,Object> passThroughParams = new HashMap<>();
  passThroughParams.put("command_results", commandResults);
  return passThroughParams;
}

// Initiate a request to open the agent
MultiModalRequestParam.UpdateParams updateParams = MultiModalRequestParam.UpdateParams.builder()
    .bizParams(MultiModalRequestParam.BizParams.builder()
        .passThroughParams(getAgentOrCommandResult())
        .build())
    .build();
conversation.requestToRespond("prompt", "", updateParams);
Agent Development
Agent Context
Agent Governance & Evolution
Store
Application Marketplace
Use Cases
Support