Java SDK

Multimodal real-time interactive architecture

Prerequisites

Enable an Alibaba Cloud Model Studio real-time multimodal interaction application to get your workspace ID, app ID, and API key.

Environment and installation

You can integrate the official Alibaba Cloud Dashscope SDK, version 2.21.16 or later, using maven or gradle.

<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>dashscope-sdk-java</artifactId>
    <version>2.21.16</version>
</dependency>

Interaction modes

The SDK supports three interaction modes: Push2Talk, Tap2Talk, and Duplex (full-duplex).

Push2Talk: Press and hold to speak, then release to stop (or click to start and click again to stop).
Tap2Talk: Click to start speaking. The SDK automatically stops recording when you finish speaking.
Duplex: Enables full-duplex interaction, allowing users to speak at any time once connected. This mode supports barge-in and is recommended for video interaction.

Audio format

Server-side integration supports only the websocket protocol.

Audio formats:
- Uplink supports pcm and opus audio for speech recognition.
- Downlink supports pcm and mp3 audio streams.

API

MultiModalDialog

Service entry class.

1. MultiModalDialog(MultiModalRequestParam param, MultiModalDialogCallback callback)

Initializes the service object and sets the dialog parameters and callback.

/**
 * Constructor initializes service options and creates a duplex communication API instance.
 *
 * @param param The dialog parameters.
 * @param callback The callback for handling events.
 */
public MultiModalDialog(MultiModalRequestParam param, MultiModalDialogCallback callback)

2. start

Starts the dialog service and triggers the onStarted callback. Note that the onStarted callback returns the dialogId.

/**
 * Starts the dialog service.
 */
public void start()

3. startSpeech

Notifies the server to start uploading audio. This method can only be called when the dialog is in the Listening state and is only required in Push-to-Talk mode.

/**
 * Starts uploading audio.
 */
public void startSpeech()

4. sendAudioData

Sends audio data to the server.

/**
 * Sends audio frame.
 *
 * @param audioFrame The audio frame.
 */
public void sendAudioData(ByteBuffer audioFrame)

5. stopSpeech

Notifies the server to stop uploading audio. This method is only required in Push-to-Talk mode.

/**
 * Stops uploading audio.
 */
public void stopSpeech()

6. interrupt

Notifies the server that the client is interrupting the current interaction to start speaking. The server then responds with RequestAccepted.

/**
 * Interrupts the current interaction.
 */
public void interrupt()

7. localRespondingStarted

Notifies the server that the client has started playing the TTS audio.

/**
 * Notifies the server that the client has started playing TTS audio.
 */
public void localRespondingStarted()

8. localRespondingEnded

Notifies the server that the client has finished playing the TTS audio.

/**
 * Notifies the server that the client has finished playing TTS audio.
 */
public void localRespondingEnded()

9. stop

Ends the current dialog turn.

/**
 * Ends the current dialog turn.
 */
public void stop()

10. getDialogState

Gets the current dialog state. The state is represented by the DialogState enum.

/**
 * Gets the current dialog state.
 *
 * @return The current dialog state.
 */
public State.DialogState getDialogState()

11. requestToRespond

Lets the client directly initiate Text-to-Speech synthesis from text or send other requests, like images, to the server.

/**
 * Requests a response from the server.
 *
 * @param type Response type
 * @param text Response text
 * @param updateParams Update parameters
 */
public void requestToRespond(String type, String text, MultiModalRequestParam.updateParams updateParams)

12. updateInfo

Updates information, such as dialog parameters.

/**
 * Updates information, such as dialog parameters.
 *
 * @param updateParams Update parameters
 */
public void updateInfo(MultiModalRequestParam.updateParams updateParams)

13. sendHeartBeat

Sends a heartbeat to prevent a 60-second timeout error. For scenarios requiring a persistent connection, call this method every 20 seconds.

/**
 * Sends a heartbeat to keep the connection alive.
 *
 */
public void sendHeartBeat()

MultiModalDialogCallback

Defines callbacks for multi-modal conversation events.

import com.alibaba.dashscope.multimodal.State;

/**
 * Abstract class representing callbacks for multi-modal conversation events.
 *
 * @author songsong.shao
 * @date 2025/4/27
 */
public abstract class MultiModalDialogCallback {

    /**
     * Called when the conversation is connected.
     */
    public abstract void onConnected();

    /**
     * Called when a conversation starts with a specific dialog ID.
     *
     * @param dialogId The unique identifier for the dialog.
     */
    public abstract void onStarted(String dialogId);

    /**
     * Called when a conversation stops with a specific dialog ID.
     *
     * @param dialogId The unique identifier for the dialog.
     */
    public abstract void onStopped(String dialogId);

    /**
     * Called when speech starts in a specific dialog.
     *
     * @param dialogId The unique identifier for the dialog.
     */
    public abstract void onSpeechStarted(String dialogId);

    /**
     * Called when speech ends in a specific dialog.
     *
     * @param dialogId The unique identifier for the dialog.
     */
    public abstract void onSpeechEnded(String dialogId);

    /**
     * Called when an error occurs during a conversation.
     *
     * @param dialogId The unique identifier for the dialog.
     * @param errorCode The error code.
     * @param errorMsg The error message.
     */
    public abstract void onError(String dialogId, String errorCode, String errorMsg);

    /**
     * Called when the conversation state changes.
     *
     * @param state The new state of the conversation.
     */
    public abstract void onStateChanged(State.DialogState state);

    /**
     * Called when speech audio data is available.
     *
     * @param audioData The audio data.
     */
    public abstract void onSpeechAudioData(ByteBuffer audioData);

    /**
     * Called when responding starts in a specific dialog.
     *
     * @param dialogId The unique identifier for the dialog.
     */
    public abstract void onRespondingStarted(String dialogId);

    /**
     * Called when responding ends in a specific dialog.
     *
     * @param dialogId The unique identifier for the dialog.
     * @param jsonObject The response content.
     */
    public abstract void onRespondingEnded(String dialogId, JsonObject jsonObject);

    /**
     * Called when responding content is available in a specific dialog.
     *
     * @param dialogId The unique identifier for the dialog.
     * @param content The response content.
     */
    public abstract void onRespondingContent(String dialogId, JsonObject content);

    /**
     * Called when speech content is available in a specific dialog.
     *
     * @param dialogId The unique identifier for the dialog.
     * @param content The speech content.
     */
    public abstract void onSpeechContent(String dialogId, JsonObject content);

    /**
     * Called when a request is accepted in a specific dialog.
     *
     * @param dialogId The unique identifier for the dialog.
     */
    public abstract void onRequestAccepted(String dialogId);

    /**
     * Called when the conversation closes.
     */
    public abstract void onClosed();
}

MultimodalRequestParam

Request parameter class. All request parameters can be configured using the builder pattern. The following tables describe each parameter.

Start request parameters

Start - Input Message

Level 1	Level 2	Level 3	Level 4	Type	Required	Description
task_group						The task group name. The value must be `aigc`.
task						The task name. The value must be `multimodal-generation`.
function						The function to call. The value must be `generation`.
model						The service name. The value must be `multimodal-dialog`.
input	workspace_id			string	Yes	The workspace ID.
	app_id			string	Yes	The ID of the application you created in the console. This ID determines which dialog system is used.
	directive			string	Yes	The directive name. The value must be `Start`.
	dialog_id			string	No	The dialog ID. If specified, the service continues the existing conversation.
parameters	upstream	type		string	Yes	The upstream type: `AudioOnly`: For audio-only calls. `AudioAndVideo`: For video uploads.
		mode		string	No	The mode used by the client. Optional values: `push2talk` `tap2talk` `duplex` The default value is `tap2talk`.
		audio_format		string	No	The audio format. Supported formats are `pcm` and `opus`. The default value is `pcm`.
	downstream	voice		string	No	The voice for the synthesized audio.
		sample_rate		int	No	The sample rate for the synthesized voice. The default is 24 kHz.
		intermediate_text		string	No	Controls which intermediate text is returned to the user: `transcript`: Returns the speech recognition result from user input. `dialog`: Returns intermediate results from the dialog system. You can specify multiple types, separated by a comma. The default value is `transcript`.
		audio_format		string	No	The audio format. Supported formats are `pcm` and `mp3`. The default value is `pcm`.
	client_info	user_id		string	Yes	The end user ID. This is used for user-specific handling.
		device	uuid	string	No	A globally unique ID for the client device. This ID must be generated by the caller and passed to the SDK.
		network	ip	string	No	The public IP address of the caller.
		location	latitude	string	No	The latitude of the caller.
			longitude	string	No	The longitude of the caller.
			city_name	string	No	The city of the caller.
	biz_params	user_defined_params		object	No	Custom parameters to pass through to the agent.
		user_defined_tokens		object	No	Custom authentication tokens to pass through to the agent.
		tool_prompts		object	No	Custom prompts to pass through to the agent.
		user_query_params		object	No	Custom parameters to pass through with the user request.
		user_prompt_params		object	No	Custom parameters to pass through with the user prompt.

RequestToRespond request parameters

Level 1	Level 2	Level 3	Type	Required	Description
input	directive		string	Yes	The directive name. The value must be `RequestToRespond`.
	dialog_id		string	Yes	The dialog ID.
	type		string	Yes	The type of interaction the service should perform: `transcript`: Converts the provided text directly to speech. `prompt`: Sends the provided text to the large model and gets a response.
	text		string	Yes	The text to process. It can be an empty string ("") but must not be null.
parameters	images		list[]	No	Information about the images to be analyzed.
	biz_params		object	No	Same as the `biz_params` in the `Start` message, used to pass custom parameters to the dialog system. The `biz_params` in a `RequestToRespond` request apply only to that request.

UpdateInfo request parameters

Level 1	Level 2	Level 3	Type	Required	Description
input	directive		string	Yes	The directive name. The value must be `UpdateInfo`.
	dialog_id		string	No	The dialog ID.
parameters	images		list[]	No	The image data.
	client_info	status	object	No	The current status of the client.
	biz_params		object	No	Same as the `biz_params` in the `Start` message, used to pass custom parameters to the dialog system. The `biz_params` in an `UpdateInfo` request apply only to that request.

Dialog state (DialogState)

The voicechat service has the following states:

IDLE("Idle"), No connection is established.
LISTENING("Listening"), The bot is listening for user input. Users can send audio.
THINKING("Thinking"), The bot is processing the user's input.
RESPONDING("Responding"), The bot is generating or sending an audio reply.

LLM dialogue output

The onRespondingContent event provides the following data.

Top-level parameter	Second-level parameter	Third-level parameter	Type	Required	Description
output	event		string	Yes	The event name, such as `RequestAccepted`.
	dialog_id		string	Yes	Unique dialogue ID.
	round_id		string	Yes	Unique ID for the current turn.
	llm_request_id		string	Yes	The request ID for the LLM call.
	text		string	Yes	Text content for display. Each streaming event provides the full, cumulative text.
	spoken		string	Yes	Text content for speech synthesis. Each streaming event provides the full, cumulative text.
	finished		bool	Yes	Indicates whether the output is complete.
	extra_info		object	No	Contains additional information. Supported fields include: `commands`: String containing commands. `agent_info`: Agent information. `tool_calls`: Information returned by plugin calls. `dialog_debug`: Dialogue debugging information. `timestamps`: Timestamps for each node in the request trace.

Call sequence diagram

Example

1. Initialize parameters

Call the builder methods of the inner classes in MultiModalRequestParam to construct the parameters.

MultiModalRequestParam params =
        MultiModalRequestParam.builder()
            .customInput(
                MultiModalRequestParam.CustomInput.builder()
                    .build())
            .upStream(
                MultiModalRequestParam.UpStream.builder()
                    .mode("push2talk")
                    .audioFormat("pcm")
                    .build())
            .downStream(
                MultiModalRequestParam.DownStream.builder()
                    .voice("longxiaochun_v2")
                    .sampleRate(48000)
                    .build())
            .dialogAttributes(
                MultiModalRequestParam.DialogAttributes.builder().prompt("你好").build())
            .clientInfo(
                MultiModalRequestParam.ClientInfo.builder()
                    .userId("1234")
                    .device(MultiModalRequestParam.ClientInfo.Device.builder().uuid("1234").build())
                    .build())
            .apiKey("sk-9d11****************************")
            .model("multimodal-dialog")
            .build();

2. Instantiate callback

public static MultiModalDialogCallback getCallback() {
    return new MultiModalDialogCallbackImpl();
  }

  public static class MultiModalDialogCallbackImpl extends MultiModalDialogCallback {
    ...
  }

3. Createa MultiModalDialog object

conversation = new MultiModalDialog(params, getCallback());

Full example

Use a build tool such as Maven or Gradle to run this code example. Add the required dependencies, including Gson, OkHttp, RxJava, Apache Log4j, Lombok, and JUnit, to the corresponding configuration file (pom.xml for Maven or build.gradle for Gradle).

Configuration
Java code

The configuration may vary depending on the version of Maven or Gradle you use. Adjust the following examples to match your environment.

Maven

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.alibaba.nls</groupId>
    <artifactId>bailian-demo</artifactId>
    <version>1.0-SNAPSHOT</version>
    <dependencies>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>dashscope-sdk-java</artifactId>
            <version>2.20.5</version>
        </dependency>
        <dependency>
            <groupId>io.reactivex.rxjava2</groupId>
            <artifactId>rxjava</artifactId>
            <version>2.2.21</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.26</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.10.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.10.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit-pioneer</groupId>
            <artifactId>junit-pioneer</artifactId>
            <version>1.9.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>io.javalin</groupId>
            <artifactId>javalin</artifactId>
            <version>4.4.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.9</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>2.0.7</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>2.0.7</version>
        </dependency>
        <!--  Required by OkHttp -->
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-stdlib-jdk8</artifactId>
            <version>1.8.21</version>
        </dependency>
        <!--  Required by OkHttp -->
        <dependency>
            <groupId>com.squareup.okio</groupId>
            <artifactId>okio</artifactId>
            <version>3.6.0</version>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>mockwebserver</artifactId>
            <version>4.12.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>logging-interceptor</artifactId>
            <version>4.12.0</version>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp-sse</artifactId>
            <version>4.12.0</version>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
            <version>4.12.0</version>
        </dependency>
        <!--  function call support  -->
        <dependency>
            <groupId>com.github.victools</groupId>
            <artifactId>jsonschema-generator</artifactId>
            <version>4.31.1</version>
        </dependency>
    </dependencies>
    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

</project>

package org.alibaba.speech.examples.speech_plus;

import com.alibaba.dashscope.multimodal.MultiModalDialog;
import com.alibaba.dashscope.multimodal.MultiModalDialogCallback;
import com.alibaba.dashscope.multimodal.MultiModalRequestParam;
import com.alibaba.dashscope.multimodal.State;
import com.alibaba.dashscope.utils.Constants;
import com.alibaba.dashscope.utils.JsonUtils;
import com.google.gson.Gson;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import org.alibaba.speech.examples.speech_plus.utils.FileWriterUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.*;

import static java.lang.Thread.sleep;

/**
 * Demonstrates how to use the MultiModalDialog class for various interaction modes,
 * including push-to-talk, tap-to-talk, duplex communication, text synthesis,
 * visual Q&A, and agent DJ functionality.
 *
 * @author songsong.shao
 * @date 2025/4/28
 */

public class MultiModalDialogTestCases {
    private static final Logger log = LoggerFactory.getLogger(MultiModalDialogTestCases.class);
    // Constants for audio processing
    private static final int AUDIO_CHUNK_SIZE = 3200; // Audio chunk size in bytes
    private static final int SLEEP_INTERVAL_MS = 100;  // Sleep interval in milliseconds
    private static final int WAIT_TIMEOUT_MS = 2000;   // Wait timeout in milliseconds
    private static final int VIDEO_FRAME_INTERVAL_MS = 500;

    // State management variables
    private static State.DialogState currentState;
    private static MultiModalDialog conversation;
    private static int enterListeningTimes = 0;
    private static FileWriterUtil fileWriterUtil;
    private static boolean vqaUseUrl = false;
    private volatile boolean isVideoStreamingActive = false;
    private Thread videoStreamingThread;

    // Configuration parameters - set these before running tests
    public static String workspaceId = "";
    public static String appId = "";
    public static String dialogId = "";
    public static String apiKey = "";
    public static String model = "";

    /**
     * Tests the push-to-talk interaction mode.
     * Flow: Set the mode to `push2talk`, wait for the Listening state, start speech, send audio, stop speech, and wait for the response.
     */
    public void testMultimodalPush2Talk() {
        log.info("############ Starting Push2Talk Test ############");

        try {
            // Build request parameters for push-to-talk mode
            MultiModalRequestParam params = buildBaseRequestParams("push2talk");
            log.debug("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation with callback
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Wait for the system to enter listening state
            waitForListeningState();

            // Start speech recognition
            conversation.startSpeech();

            // Send audio data from file
            sendAudioFromFile("../../../../sample-data/1_plus_1.wav");

            // Stop speech recognition
            conversation.stopSpeech();

            // Wait for conversation completion
            waitForConversationCompletion(2);

            // Clean up
            stopConversation();

        } catch (Exception e) {
            log.error("Error in push2talk test: ", e);
        } finally {
            log.info("############ Push2Talk Test Completed ############");
        }
    }

    /**
     * Tests the tap-to-talk interaction mode.
     * Flow: Set the mode to `tap2talk`, wait for the Listening state, manually start speech, send audio, and wait for the response.
     */
    public void testMultimodalTap2Talk() {
        log.info("############ Starting Tap2Talk Test ############");

        try {
            // Build request parameters for tap-to-talk mode
            MultiModalRequestParam params = buildBaseRequestParams("tap2talk");
            log.debug("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Wait for listening state
            waitForListeningState();

            // Manually start speech (required for tap2talk mode)
            conversation.startSpeech();

            // Send audio data
            sendAudioFromFile("../../../../sample-data/1_plus_1.wav");

            // Wait for conversation completion
            waitForConversationCompletion(2);

            // Clean up
            stopConversation();

        } catch (Exception e) {
            log.error("Error in tap2talk test: ", e);
        } finally {
            log.info("############ Tap2Talk Test Completed ############");
        }
    }

    /**
     * Tests the duplex communication interaction mode.
     * Flow: Set the mode to `duplex`, wait for the Listening state, send audio, and wait for the response.
     */
    public void testMultimodalDuplex() {
        log.info("############ Starting Duplex Test ############");

        try {
            // Build request parameters for duplex mode
            MultiModalRequestParam params = buildBaseRequestParams("duplex");
            log.info("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Wait for listening state
            waitForListeningState();

            // Send audio data directly (no manual start needed for duplex)
            sendAudioFromFile("../../../../sample-data/1_plus_1.wav");

            // Wait for conversation completion
            waitForConversationCompletion(2);

            // Clean up
            stopConversation();

        } catch (Exception e) {
            log.error("Error in duplex test: ", e);
        } finally {
            log.info("############ Duplex Test Completed ############");
        }
    }

    /**
     * Tests the text synthesis feature.
     * Flow: Send text for TTS and save the synthesized audio.
     */
    public void testMultimodalTextSynthesizer() {
        log.info("############ Starting Text Synthesizer Test ############");

        try {
            // Initialize file writer for audio output
            fileWriterUtil = new FileWriterUtil();

            // Build request parameters
            MultiModalRequestParam params = buildBaseRequestParams("push2talk");
            log.debug("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Wait for listening state
            waitForListeningState();

            // Create output file for synthesized audio
            fileWriterUtil.createFile("./test.pcm");

            // Request text synthesis
            String textToSynthesize = "Happiness is a skill. It's the inner peace you feel when you let go of wanting too much.";
            conversation.requestToRespond("transcript", textToSynthesize, null);

            // Wait for synthesis completion
            waitForConversationCompletion(2);

            // Finalize audio file
            fileWriterUtil.finishWriting();

            // Clean up
            conversation.stop();

        } catch (IOException e) {
            log.error("File operation error in text synthesizer test: ", e);
        } catch (Exception e) {
            log.error("Error in text synthesizer test: ", e);
        } finally {
            log.info("############ Text Synthesizer Test Completed ############");
        }
    }

    /**
     * Tests the visual Q&A feature by using a URL-based image.
     * Flow: Send a visual Q&A request, receive a `visual_qa` command, send an image list, and get the response.
     */
    public void testMultimodalVQA() {
        log.info("############ Starting VQA Test (URL-based) ############");

        try {
            vqaUseUrl = true;

            // Build request parameters
            MultiModalRequestParam params = buildBaseRequestParams("push2talk");
            log.debug("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Wait for listening state
            waitForListeningState();

            // Send VQA request
            conversation.requestToRespond("prompt", "Take a picture and tell me what you see.", null);

            // Wait for VQA processing completion
            waitForConversationCompletion(3);

            // Clean up
            stopConversation();

        } catch (Exception e) {
            log.error("Error in VQA test: ", e);
        } finally {
            log.info("############ VQA Test Completed ############");
        }
    }

    /**
     * Tests the visual Q&A feature by using a Base64-encoded image.
     * Flow: Send a visual Q&A request, receive a `visual_qa` command, send the Base64-encoded image, and get the response.
     */
    public void testMultimodalVQABase64() {
        log.info("############ Starting VQA Test (Base64-based) ############");

        try {
            vqaUseUrl = false;

            // Build request parameters
            MultiModalRequestParam params = buildBaseRequestParams("push2talk");
            log.debug("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Wait for listening state
            waitForListeningState();

            // Send VQA request
            conversation.requestToRespond("prompt", "Take a picture and tell me what you see.", null);

            // Wait for VQA processing completion
            waitForConversationCompletion(3);

            // Clean up
            stopConversation();

        } catch (Exception e) {
            log.error("Error in VQA Base64 test: ", e);
        } finally {
            log.info("############ VQA Base64 Test Completed ############");
        }
    }

    /**
     * Tests the brightness adjustment feature.
     * Flow: Send a brightness adjustment request and process the response.
     */
    public void testMultimodalAdjustBrightness() {
        log.info("############ Starting Brightness Adjustment Test ############");

        try {
            vqaUseUrl = true;

            // Build request parameters
            MultiModalRequestParam params = buildBaseRequestParams("push2talk");
            log.debug("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Wait for listening state
            waitForListeningState();

            // Send brightness adjustment request
            conversation.requestToRespond("prompt", "Increase the brightness to 10.", null);

            // The following code block demonstrates creating a status update. It is not used in this test.
            JsonObject status = new JsonObject();
            JsonObject bluetoothAnnouncement = new JsonObject();
            bluetoothAnnouncement.addProperty("status", "stopped");
            status.add("bluetooth_announcement", bluetoothAnnouncement);

            MultiModalRequestParam.UpdateParams updateParams = MultiModalRequestParam.UpdateParams.builder()
                    .clientInfo(MultiModalRequestParam.ClientInfo.builder()
                            .status(status)
                            .build())
                    .build();

            // Wait for processing completion
            waitForConversationCompletion(2);

            log.info("############ Before stopping conversation ############");

            // Clean up
            stopConversation();

        } catch (Exception e) {
            log.error("Error in brightness adjustment test: ", e);
        } finally {
            log.info("############ Brightness Adjustment Test Completed ############");
        }
    }

    /**
     * Tests the Agent DJ feature.
     * Flow: Send a radio station request and process the response.
     */
    public void testMultimodalAgentDJ() {
        log.info("############ Starting Agent DJ Test ############");

        try {
            vqaUseUrl = false;

            // Build request parameters
            MultiModalRequestParam params = buildBaseRequestParams("push2talk");
            log.debug("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Wait for listening state
            waitForListeningState();

            // Send DJ request
            conversation.requestToRespond("prompt", "Turn on the news radio station.", null);

            // Wait for DJ processing completion
            waitForConversationCompletion(3);

            // Clean up
            stopConversation();

        } catch (Exception e) {
            log.error("Error in Agent DJ test: ", e);
        } finally {
            log.info("############ Agent DJ Test Completed ############");
        }
    }

    /**
     * Tests the LiveAI feature with real-time video streaming and audio interaction.
     * Flow: Set the mode to duplex, start video streaming, wait for the Listening state, connect the video channel, send audio queries, and then process responses that have visual context.
     */
    public void testMultimodalLiveAI() {
        log.info("############ Starting LiveAI Test ############");

        try {
            // Build request parameters for duplex mode
            MultiModalRequestParam params = buildBaseRequestParams("duplex");
            log.info("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Start send video frame loop
            startVideoFrameStreaming();
            // Wait for listening state
            waitForListeningState();
            conversation.sendHeartBeat();
            // Sends a request to connect the video channel. The service responds with a `switch_video_call_success` command.
            conversation.requestToRespond("prompt", "", connectVideoChannelRequest());

            // Send audio data directly (no manual start needed for duplex)
            sendAudioFromFile("./src/main/resources/what_in_picture.wav");

            // Wait for listening state twice
            waitForListeningState();

            sendAudioFromFile("./src/main/resources/what_color.wav");

            // Wait for conversation completion
            waitForConversationCompletion(3);

            // Clean up
            stopConversation();

            isVideoStreamingActive = false;
            if (videoStreamingThread != null && videoStreamingThread.isAlive()) {
                videoStreamingThread.interrupt();
            }

        } catch (Exception e) {
            log.error("Error in LiveAI test: ", e);
        } finally {
            log.info("############ LiveAI Test Completed ############");
        }
    }

    // ==================== Helper Methods ====================

    /**
     * Builds base request parameters with common configuration.
     * @param mode The interaction mode (push2talk, tap2talk, duplex)
     * @return Configured MultiModalRequestParam
     */
    private MultiModalRequestParam buildBaseRequestParams(String mode) {
        return MultiModalRequestParam.builder()
                .customInput(
                        MultiModalRequestParam.CustomInput.builder()
                                .workspaceId(workspaceId)
                                .appId(appId)
                                .build())
                .upStream(
                        MultiModalRequestParam.UpStream.builder()
                                .mode(mode)
                                .audioFormat("pcm")
                                .build())
                .downStream(
                        MultiModalRequestParam.DownStream.builder()
                                .voice("longxiaochun_v2")
                                .sampleRate(48000)
                                .build())
                .clientInfo(
                        MultiModalRequestParam.ClientInfo.builder()
                                .userId("1234")
                                .device(MultiModalRequestParam.ClientInfo.Device.builder()
                                        .uuid("1234")
                                        .build())
                                .build())
                .apiKey(apiKey)
                .model(model)
                .build();
    }

    /**
     * Builds a video channel connection request for LiveAI.
     * This request establishes the video streaming channel that provides visual context for AI responses.
     *
     * @return UpdateParams containing video channel connection configuration
     */
    private MultiModalRequestParam.UpdateParams connectVideoChannelRequest(){

        Map<String, String> video = new HashMap<>();
        video.put("action", "connect");
        video.put("type", "voicechat_video_channel");
        ArrayList<Map<String, String>> videos = new ArrayList<>();
        videos.add(video);
        MultiModalRequestParam.BizParams bizParams = MultiModalRequestParam.BizParams.builder().videos(videos).build();

        return MultiModalRequestParam.UpdateParams.builder().bizParams(bizParams).build();
    }

    /**
     * Waits for the system to enter the Listening state.
     */
    private void waitForListeningState() {
        while (currentState != State.DialogState.LISTENING) {
            try {
                Thread.sleep(SLEEP_INTERVAL_MS);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                throw new RuntimeException("Interrupted while waiting for listening state", e);
            }
        }
        log.info("System entered listening state");
    }

    /**
     * Sends audio data from a file to the conversation.
     * @param filePath Path to the audio file
     */
    private void sendAudioFromFile(String filePath) {
        File audioFile = new File(filePath);

        if (!audioFile.exists()) {
            log.error("Audio file not found: {}", filePath);
            return;
        }

        try (AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(audioFile)) {
            byte[] audioBuffer = new byte[AUDIO_CHUNK_SIZE];
            int bytesRead;
            int totalBytesRead = 0;

            log.info("Starting to send audio data from: {}", filePath);

            // Read and send audio data in chunks
            while ((bytesRead = audioInputStream.read(audioBuffer)) != -1) {
                totalBytesRead += bytesRead;

                // Send audio chunk to conversation
                conversation.sendAudioData(ByteBuffer.wrap(audioBuffer, 0, bytesRead));

                // Add small delay to simulate real-time audio streaming
                Thread.sleep(SLEEP_INTERVAL_MS);
            }

            log.info("Finished sending audio data. Total bytes sent: {}", totalBytesRead);

        } catch (Exception e) {
            log.error("Error sending audio from file: {}", filePath, e);
        }
    }

    /**
     * Waits for the conversation to complete.
     * @param expectedListeningTimes Expected number of listening state entries
     */
    private void waitForConversationCompletion(int expectedListeningTimes) {
        while (enterListeningTimes < expectedListeningTimes) {
            try {
                Thread.sleep(WAIT_TIMEOUT_MS);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                throw new RuntimeException("Interrupted while waiting for conversation completion", e);
            }
        }
        log.info("Conversation completed after {} listening cycles", expectedListeningTimes);
    }

    /**
     * Stops the conversation and cleans up resources.
     */
    private void stopConversation() {
        try {
            if (conversation != null) {
                conversation.stop();
                Thread.sleep(1000); // Allow time for cleanup
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            log.warn("Interrupted while stopping conversation");
        }
    }

    /**
     * Returns the callback implementation for handling conversation events.
     * @return MultiModalDialogCallback instance
     */
    public static MultiModalDialogCallback getCallback() {
        return new MultiModalDialogCallbackImpl();
    }

    /**
     * Implementation of MultiModalDialogCallback for handling various conversation events.
     */
    public static class MultiModalDialogCallbackImpl extends MultiModalDialogCallback {

        @Override
        public void onConnected() {
            log.info("WebSocket connection established");
        }

        @Override
        public void onStarted(String dialogId) {
            log.info("Dialog started with ID: {}", dialogId);
        }

        @Override
        public void onStopped(String dialogId) {
            log.info("Dialog stopped with ID: {}", dialogId);
        }

        @Override
        public void onSpeechStarted(String dialogId) {
            log.info("Speech recognition started for dialog: {}", dialogId);
        }

        @Override
        public void onSpeechEnded(String dialogId) {
            log.info("Speech recognition ended for dialog: {}", dialogId);
        }

        @Override
        public void onError(String dialogId, String errorCode, String errorMsg) {
            log.error("Error occurred - Dialog: {}, Code: {}, Message: {}", dialogId, errorCode, errorMsg);
            enterListeningTimes++; // Allows the test to terminate on error
        }

        @Override
        public void onStateChanged(State.DialogState state) {
            log.info("Dialog state changed to: {}", state);
            currentState = state;

            if (currentState == State.DialogState.LISTENING) {
                enterListeningTimes++;
                log.info("Entered listening state {} times", enterListeningTimes);
            }
        }

        @Override
        public void onSpeechAudioData(ByteBuffer audioData) {
            try {
                // Write audio data to file if file writer is available
                if (fileWriterUtil != null) {
                    fileWriterUtil.Writing(audioData);
                }
            } catch (IOException e) {
                log.error("Error writing audio data to file: ", e);
                throw new RuntimeException("Failed to write audio data", e);
            }
        }

        @Override
        public void onRespondingStarted(String dialogId) {
            log.info("Response generation started for dialog: {}", dialogId);
            if (conversation != null) {
                conversation.localRespondingStarted();
            }
        }

        @Override
        public void onRespondingEnded(String dialogId, JsonObject jsonObject) {
            log.info("Response generation ended for dialog: {}", dialogId);
            if (conversation != null) {
                conversation.localRespondingEnded();
            }
        }

        @Override
        public void onRespondingContent(String dialogId, JsonObject content) {
            log.info("Response content received - Dialog: {}, Content: {}", dialogId, content);

            // Handle visual Q&A commands
            handleVisualQACommands(content);
        }

        @Override
        public void onSpeechContent(String dialogId, JsonObject content) {
            log.info("Speech content received - Dialog: {}, Content: {}", dialogId, content);
        }

        @Override
        public void onRequestAccepted(String dialogId) {
            log.info("Request accepted for dialog: {}", dialogId);
        }

        @Override
        public void onClosed() {
            log.info("Connection closed");
            enterListeningTimes++; // Increment to trigger test completion
        }

        /**
         * Handles visual Q&A commands from the response content.
         * @param content Response content containing potential commands
         */
        private void handleVisualQACommands(JsonObject content) {
            if (!content.has("extra_info")) {
                return;
            }

            JsonObject extraInfo = content.getAsJsonObject("extra_info");
            if (!extraInfo.has("commands")) {
                return;
            }

            try {
                String commandsStr = extraInfo.get("commands").getAsString();
                log.info("Processing commands: {}", commandsStr);

                JsonArray commands = new Gson().fromJson(commandsStr, JsonArray.class);

                for (JsonElement command : commands) {
                    JsonObject commandObj = command.getAsJsonObject();

                    if (commandObj.has("name")) {
                        String commandName = commandObj.get("name").getAsString();

                        if ("visual_qa".equals(commandName)) {
                            log.info("Visual Q&A command detected - triggering image capture");

                            // Send mock image data for visual Q&A
                            MultiModalRequestParam.UpdateParams updateParams =
                                    MultiModalRequestParam.UpdateParams.builder()
                                            .images(getMockImageRequest())
                                            .build();

                            if (conversation != null) {
                                conversation.requestToRespond("prompt", "", updateParams);
                            }
                        }
                    }
                }
            } catch (Exception e) {
                log.error("Error processing visual Q&A commands: ", e);
            }
        }
    }

    /**
     * Creates mock image data for testing purposes.
     * @return A list of image objects, which can be URL-based or Base64-encoded depending on the vqaUseUrl flag.
     */
    public static List<Object> getMockImageRequest() {
        List<Object> images = new ArrayList<>();

        try {
            JsonObject imageObject = new JsonObject();

            if (vqaUseUrl) {
                // Use URL-based image
                imageObject.addProperty("type", "url");
                imageObject.addProperty("value", "https://help-static-aliyun-doc.aliyuncs.com/assets/img/zh-CN/7043267371/p909896.png");
                imageObject.addProperty("bucket", "bucketName");
                imageObject.add("extra", new JsonObject());
            } else {
                // Use Base64-encoded image
                imageObject.addProperty("type", "base64");
                imageObject.addProperty("value", getLocalImageBase64());
            }

            images.add(imageObject);

        } catch (Exception e) {
            log.error("Error creating mock image data: ", e);
        }

        return images;
    }

    /**
     * Converts a local image file to a Base64 string.
     * @return Base64-encoded image string
     */
    public static String getLocalImageBase64() {
        String imagePath = "./src/main/resources/jpeg-bridge.jpg";

        try (FileInputStream fileInputStream = new FileInputStream(new File(imagePath))) {
            byte[] imageBytes = new byte[fileInputStream.available()];
            fileInputStream.read(imageBytes);

            String base64Image = Base64.getEncoder().encodeToString(imageBytes);
            log.info("Successfully converted image to Base64, size: {} bytes", imageBytes.length);

            return base64Image;

        } catch (IOException e) {
            log.error("Error converting image to Base64: {}", imagePath, e);
            return null;
        }
    }

    /**
     * Starts continuous video frame streaming for LiveAI.
     */
    private void startVideoFrameStreaming() {
        log.info("Starting continuous video frame streaming for LiveAI...");

        vqaUseUrl = false;
        isVideoStreamingActive = true;

        videoStreamingThread = new Thread(() -> {
            try {
                while (isVideoStreamingActive && !Thread.currentThread().isInterrupted()) {
                    Thread.sleep(VIDEO_FRAME_INTERVAL_MS);

                    MultiModalRequestParam.UpdateParams videoUpdate =
                            MultiModalRequestParam.UpdateParams.builder()
                                    .images(getMockImageRequest())
                                    .build();

                    if (conversation != null && isVideoStreamingActive) {
                        conversation.updateInfo(videoUpdate);
                        log.debug("Video frame sent to LiveAI");
                    }
                }
            } catch (InterruptedException e) {
                log.info("Video streaming thread interrupted - stopping video stream");
                Thread.currentThread().interrupt();
            } catch (Exception e) {
                log.error("Error in video streaming thread: ", e);
            } finally {
                log.info("Video streaming thread terminated");
            }
        });

        videoStreamingThread.setDaemon(true);
        videoStreamingThread.setName("LiveAI-VideoStreaming");
        videoStreamingThread.start();

        log.info("Video streaming thread started successfully");
    }

    /**
     * Main method to run the test cases.
     * Configure the required parameters before running.
     */
    public static void main(String[] args) {
        log.info("############ Initializing multi-modal dialog tests ############");

        // Configure WebSocket API URL
        Constants.baseWebsocketApiUrl = "wss://dashscope.aliyuncs.com/api-ws/v1/inference";
        log.info("WebSocket API URL: {}", Constants.baseWebsocketApiUrl);

        // Reset state variables
        enterListeningTimes = 0;

        // TODO: Configure your test parameters.
        MultiModalDialogTestCases.workspaceId = "";
        MultiModalDialogTestCases.appId = "";
        MultiModalDialogTestCases.apiKey = "";
        MultiModalDialogTestCases.model = "multimodal-dialog";

        // Validate configuration
        if (apiKey.isEmpty() || workspaceId.isEmpty() || appId.isEmpty()) {
            log.error("Please configure workspaceId, appId, and apiKey before running tests");
            return;
        }

        // Create test instance and run a specific test
        MultiModalDialogTestCases testCases = new MultiModalDialogTestCases();

        try {
            // Run the desired test case
            testCases.testMultimodalPush2Talk();

            // Uncomment other test cases as needed:
            // testCases.testMultimodalTap2Talk();
            // testCases.testMultimodalDuplex();
            // testCases.testMultimodalTextSynthesizer();
            // testCases.testMultimodalVQA();
            // testCases.testMultimodalVQABase64();
            // testCases.testMultimodalAdjustBrightness();
            // testCases.testMultimodalAgentDJ();
            // testCases.testMultimodalLiveAI();

        } catch (Exception e) {
            log.error("Error running test cases: ", e);
        }

        log.info("############ Multi-modal dialog tests completed ############");
    }
}

Advanced SDK usage

VQA interaction

VQA (Visual Question Answering) combines images and voice for multimodal interaction, allowing you to send images during a conversation. A voice or text request with a photo-taking intent triggers the "visual_qa" command. When you receive this command, send an image as either a URL or a base64-encoded string. The image size must be less than 180 KB.

Initiate a photo request after establishing a connection.

@Test
  void testMultimodalVQA() {
    /*
      Step 1: Send "What's in front of me?", and onRespondingContent returns the visual_qa command.
      Step 2: Send the image list.
      Step 3: Receive the conversation result for the image.
      */
    System.out.println("############ Start Test VQA ############");
    vqaUseUrl = true;

    log.debug("params: {}", JsonUtils.toJson(params));

    conversation = new MultiModalDialog(params, getCallback());

    conversation.start();
    while (currentState != State.DialogState.LISTENING) {
      try {
        sleep(100);
      } catch (InterruptedException e) {
        throw new RuntimeException(e);
      }
    }
    conversation.requestToRespond("prompt","Take a picture and tell me what you see.",null);
    // Wait for the interaction flow.
    while (enterListeningTimes < 3) {
      try {
        sleep(2000);
      } catch (InterruptedException e) {
        throw new RuntimeException(e);
      }
    }
    conversation.stop();
    try {
      sleep(1000);
    } catch (InterruptedException e) {
      throw new RuntimeException(e);
    }
    System.out.println("############ End Test VQA ############");
  }

Handle the "visual_qa" command and upload the photo.

@Override
    public void onRespondingContent(String dialogId, JsonObject content) {
      log.info("onRespondingContent: {}, {}", dialogId, content);
      if (content.has("extra_info")) {
        JsonObject extraInfo = content.getAsJsonObject("extra_info");
        if (extraInfo.has("commands")) {
          String commandsStr = extraInfo.get("commands").getAsString();
          log.info("commandsStr: {}", commandsStr);
          //"[{\"name\":\"visual_qa\",\"params\":[{\"name\":\"shot\",\"value\":\"拍照看看\",\"normValue\":\"True\"}]}]"
          JsonArray commands = new Gson().fromJson(commandsStr, JsonArray.class);
          for (JsonElement command : commands) {
            JsonObject commandObj = command.getAsJsonObject();
            if (commandObj.has("name")) {
              String commandStr = commandObj.get("name").getAsString();
              if (commandStr.equals("visual_qa")) {
                log.info("Taking a picture!");
                MultiModalRequestParam.UpdateParams updateParams = MultiModalRequestParam.UpdateParams.builder()
                        .images(getMockOSSImage())
                        .build();
                conversation.requestToRespond("prompt","",updateParams);
              }
            }
          }
        }
      }
    }

// Upload the photo
public static List<Object> getMockOSSImage() {
    JsonObject imageObject = new JsonObject();
    JsonObject extraObject = new JsonObject();
    List<Object> images = new ArrayList<>();
    try{
      if (vqaUseUrl){
        imageObject.addProperty("type", "url");
        imageObject.addProperty("value", "https://help-static-aliyun-doc.aliyuncs.com/assets/img/zh-CN/7043267371/p909896.png");
        imageObject.addProperty("bucket", "bucketName");
        imageObject.add("extra", extraObject);
      }else {
        imageObject.addProperty("type", "base64");
        imageObject.addProperty("value", getLocalImageBase64());
      }
      images.add(imageObject);

    }catch (Exception e){
      e.printStackTrace();
    }
    return images;
  }

TTS synthesis

The SDK supports server-side audio synthesis directly from text. You need to send a requestToRespond request when the client is in the Listening state. If the current state is not Listening, you must first call the Interrupt API to interrupt the current playback.

  @Test
  void testMultimodalTextSynthesizer() {
    /*
      Step 1: Send text for TTS synthesis.
      Step 2: Save the synthesized TTS audio.
      */
    System.out.println("############ Start Test TextSynthesizer ############");

    conversation = new MultiModalDialog(params, getCallback());

    conversation.start();
    while (currentState != State.DialogState.LISTENING) {
      try {
        sleep(100);
      } catch (InterruptedException e) {
        throw new RuntimeException(e);
      }
    }
    try {
      fileWriterUtil.createFile("./test.pcm");
    } catch (IOException e) {
      log.error(e.getMessage());
      throw new RuntimeException(e);
    }
    conversation.requestToRespond("transcript","Happiness is a skill; it's the inner peace that comes from letting go of excessive external desires.",null);

    // Wait for the interaction flow.
    while (enterListeningTimes < 2) {
      try {
        sleep(2000);
      } catch (InterruptedException e) {
        throw new RuntimeException(e);
      }
    }
    try {
      fileWriterUtil.finishWriting();
    } catch (IOException e) {
      log.error(e.getMessage());
      throw new RuntimeException(e);
    }
    conversation.stop();

    System.out.println("############ End Test TextSynthesizer ############");
  }

Request LiveAI via WebSocket

LiveAI, an official agent from Model Studio for multimodal interaction, enables video calls. Implement video calls by sending a sequence of images through the Java SDK. We recommend using Real-Time Communication (RTC) to transmit video and audio between your server and client (web page or app). Then, send the video frames captured by the server to the SDK at a rate of one frame every 500 ms, while maintaining a real-time audio input stream. Note: When sending images to LiveAI, only base64 encoding is supported, and each image must be smaller than 180 KB.

LiveAI call sequence

Key code examples

For the complete code, see the GitHub sample code.

public void testMultimodalLiveAI() {
        log.info("############ Starting LiveAI Test ############");

        try {
            // Build request parameters for duplex mode
            MultiModalRequestParam params = buildBaseRequestParams("duplex");
            log.info("Request parameters: {}", JsonUtils.toJson(params));

            // Initialize conversation
            conversation = new MultiModalDialog(params, getCallback());
            conversation.start();

            // Start send video frame loop
            startVideoFrameStreaming();
            // Wait for listening state
            waitForListeningState();
            conversation.sendHeartBeat();
            // Send video channel connect request, will response command : switch_video_call_success
            conversation.requestToRespond("prompt", "", connectVideoChannelRequest());

            // Send audio data directly (no manual start needed for duplex)
            sendAudioFromFile("./src/main/resources/what_in_picture.wav");

            // Wait for listening state twice
            waitForListeningState();

            sendAudioFromFile("./src/main/resources/what_color.wav");

            // Wait for conversation completion
            waitForConversationCompletion(3);

            // Clean up
            stopConversation();

            isVideoStreamingActive = false;
            if (videoStreamingThread != null && videoStreamingThread.isAlive()) {
                videoStreamingThread.interrupt();
            }

        } catch (Exception e) {
            log.error("Error in LiveAI test: ", e);
        } finally {
            log.info("############ LiveAI Test Completed ############");
        }
    }

/**
     * Build video channel connection request for LiveAI
     *
     * @return UpdateParams containing video channel connection configuration
     */
    private MultiModalRequestParam.UpdateParams connectVideoChannelRequest(){

        Map<String, String> video = new HashMap<>();
        video.put("action", "connect");
        video.put("type", "voicechat_video_channel");
        ArrayList<Map<String, String>> videos = new ArrayList<>();
        videos.add(video);
        MultiModalRequestParam.BizParams bizParams = MultiModalRequestParam.BizParams.builder().videos(videos).build();

        return MultiModalRequestParam.UpdateParams.builder().bizParams(bizParams).build();
    }

/**
     * Start continuous video frame streaming for LiveAI
     */
    private void startVideoFrameStreaming() {
        log.info("Starting continuous video frame streaming for LiveAI...");

        vqaUseUrl = false;
        isVideoStreamingActive = true;

        videoStreamingThread = new Thread(() -> {
            try {
                while (isVideoStreamingActive && !Thread.currentThread().isInterrupted()) {
                    Thread.sleep(VIDEO_FRAME_INTERVAL_MS);

                    MultiModalRequestParam.UpdateParams videoUpdate =
                            MultiModalRequestParam.UpdateParams.builder()
                                    .images(getMockImageRequest())
                                    .build();

                    if (conversation != null && isVideoStreamingActive) {
                        conversation.updateInfo(videoUpdate);
                        log.debug("Video frame sent to LiveAI");
                    }
                }
            } catch (InterruptedException e) {
                log.info("Video streaming thread interrupted - stopping video stream");
                Thread.currentThread().interrupt();
            } catch (Exception e) {
                log.error("Error in video streaming thread: ", e);
            } finally {
                log.info("Video streaming thread terminated");
            }
        });

        videoStreamingThread.setDaemon(true);
        videoStreamingThread.setName("LiveAI-VideoStreaming");
        videoStreamingThread.start();

        log.info("Video streaming thread started successfully");
    }

Use custom prompt variables

In the console, navigate to your project's Prompt settings to configure custom variables.

The following example defines a user_name field that represents a user nickname. The user_name variable is then inserted into the Prompt as the placeholder ${user_name}. The Custom Variable button is located in the Available Variables area at the top of the prompt configuration page.

Set variables in your code.

For example, set "user_name" = "Alice".

// Pass biz_params.user_prompt_params when building the request parameters.
MultiModalRequestParam.builder()
        .bizParams(
                MultiModalRequestParam.BizParams.builder()
                        .userPromptParams(setUserPromptParams())
                        .build())
        .apiKey(apiKey)
        .model(model)
        .build();
private HashMap<String, String> setUserPromptParams() {
        HashMap<String, String> promptSettings = new HashMap<>();
        promptSettings.put("user_name", "Dami");
        return promptSettings;
    }

Request a response.

The agent uses Dear Rice to address the user in its response, for example, "Dear Rice, what is the weather like where you are today?", which indicates that the user_name variable was successfully passed and took effect in the Prompt.

Real-time ASR correction

During a conversation, ASR (Automatic Speech Recognition) results may contain errors or unexpected outputs. In addition to configuring hotwords, you can use the real-time correction feature to upload a word list and fix these errors immediately.

Parameter description

Configure the correction word list by setting the AsrPostProcessing parameter in UpStream.

Parameter	Level 1 parameter	Level 2 parameter	Type	Description
AsrPostProcessing			Object	The ASR correction word list.
	ReplaceWord[]		List	A list of replacement rules. Each `ReplaceWord` object defines a rule for a word or phrase.
		source	String	The text to be replaced.
		target	String	The replacement text.
		match_mode	String	The match mode. Default: `exact`. exact: Exact match. The entire recognized text must be identical to the source value. partial: Partial match. A match occurs if the source value is found as a substring within the recognized text.

To use this feature, your Java SDK version must be 2.21.14 or later.

// Build the AsrPostProcessing object.
MultiModalRequestParam.UpStream.AsrPostProcessing asrPostProcessing =
MultiModalRequestParam.UpStream.AsrPostProcessing
.builder()
.replaceWords(Collections.singletonList(
        ReplaceWord.builder()
        .source("1 plus 1")
        .target("one plus one")
        .matchMode("partial")
        .build()
        ))
.build();
// When creating the dialog request, pass asrPostProcessing to UpStream.AsrPostProcessing.

More examples

For more examples, see the GitHub sample code.

​Multimodal real-time interactive architecture

​Prerequisites

​Environment and installation

​Interaction modes

​Audio format

​API

​MultiModalDialog

​1. MultiModalDialog(MultiModalRequestParam param, MultiModalDialogCallback callback)

​2. start

​3. startSpeech

​4. sendAudioData

​5. stopSpeech

​6. interrupt

​7. localRespondingStarted

​8. localRespondingEnded

​9. stop

​10. getDialogState

​11. requestToRespond

​12. updateInfo

​13. sendHeartBeat

​MultiModalDialogCallback

​MultimodalRequestParam

​Start request parameters

​RequestToRespond request parameters

​UpdateInfo request parameters

​Dialog state (DialogState)

​LLM dialogue output

​Call sequence diagram

​Example

​1. Initialize parameters

​2. Instantiate callback

​3. Createa MultiModalDialog object

​Full example

​Advanced SDK usage

​VQA interaction

​TTS synthesis

​Request LiveAI via WebSocket

​Use custom prompt variables

​Real-time ASR correction

​More examples

Multimodal real-time interactive architecture

Prerequisites

Environment and installation

Interaction modes

Audio format

API

MultiModalDialog

1. MultiModalDialog(MultiModalRequestParam param, MultiModalDialogCallback callback)

2. start

3. startSpeech

4. sendAudioData

5. stopSpeech

6. interrupt

7. localRespondingStarted

8. localRespondingEnded

9. stop

10. getDialogState

11. requestToRespond

12. updateInfo

13. sendHeartBeat

MultiModalDialogCallback

MultimodalRequestParam

Start request parameters

RequestToRespond request parameters

UpdateInfo request parameters

Dialog state (DialogState)

LLM dialogue output

Call sequence diagram

Example

1. Initialize parameters

2. Instantiate callback

3. Createa MultiModalDialog object

Full example

Advanced SDK usage

VQA interaction

TTS synthesis

Request LiveAI via WebSocket

Use custom prompt variables

Real-time ASR correction

More examples