Skip to main content

iOS Lite SDK

This topic describes how to use the real-time multimodal interaction iOS Lite SDK provided by Alibaba Cloud Model Studio for Large Language Models. It covers SDK download and installation, key interfaces, and code examples.

The MultiModalDialog SDK is a software development kit (SDK) from Alibaba Cloud’s Tongyi team. It enables end-to-end, real-time multimodal interaction for audio and video. By integrating this SDK with Qwen large language models and various backend agents, you can support voice conversations, weather queries, music playback, news updates, and other capabilities. It also supports large model–powered dialogue using video and images.

Architecture of the Multimodal Real-Time Interaction Service

Multimodal Real-time Interaction Service Architecture Diagram

Prerequisites

Enable an Alibaba Cloud Model Studio real-time multimodal interaction application to get your workspace ID, app ID, and API key. The multimodal dialog iOS Lite SDK uses WebSocket connections exclusively to interact with the server. It supports AudioOnly mode for voice conversations.
  • Audio interaction: Uses WebSocket connections. These connections establish quickly and require low performance overhead.
    • Audio format for WebSocket links:
      • Uplink: Supports PCM and Opus formats for speech recognition.
      • Downlink: Supports PCM and MP3 audio streams.

Interaction Modes

The SDK supports three interaction modes: Push2Talk, Tap2Talk, and Duplex (full-duplex).
  • Push2Talk: Press and hold to speak. Release to stop. Alternatively, tap to start and tap again to stop.
  • Tap2Talk: Tap to start speaking. The system automatically detects when you finish speaking.
  • Duplex: Full-duplex interaction. You can speak at any time after connecting. Voice interruption is supported. Note: Full-duplex interaction requires echo cancellation (AEC) on the client side. The Lite SDK does not include an AEC module. You can integrate your own AEC module or use the full-featured Alibaba Cloud Model Studio iOS SDK for Mobile.

Environment and Dependencies

  • Import the SDK dashscope-lite-ios-demo-1.0.1.zip
    • DashscopeLite.framework: The Lite SDK for multimodal dialog.
    • Other dependencies:
      • 'SocketRocket', '~> 0.7.1'
  • Connect to the Alibaba Cloud Model Studio environment
    • The multimodal dialog service runs on Alibaba Cloud Model Studio. Authentication uses your Alibaba Cloud Model Studio API key. Create an API key in the Alibaba Cloud Model Studio console. For security, you can also generate short-term tokens on your server and send them to the client. See short-term token.

Interface Reference

MultimodalDialog

Service entry class

1 MultiModalDialog

Initialize the service object. Set dialog parameters and callbacks. 
/// Initialize the multimodal dialog manager.
/// - Parameters:
///   - url: Service URL
///   - workSpaceId: Workspace ID. Get it from the Alibaba Cloud Model Studio console.
///   - appId: App ID. Configure it in the multimodal dialog console.
///   - mode: Dialog mode (tap2talk, push2talk, or duplex)
public init(url: String?, workSpaceId: String, appId:String ,mode: DialogMode)

2 start

Start the dialog service. Returns the onConversationStarted callback. 
/// Start a dialog request and connect to the dialog service.
/// - Parameters:
///   - apiKey: API key
///   - params: Multimodal request parameters
///   - completion: Completion callback. Returns whether the connection succeeded and any error message.
public func start(apiKey: String, params: MultiModalRequestParam, completion: @escaping (Bool, TYError?) -> Void )

3startSpeech

Notify the server to start uploading audio. Call this only when the state is Listening. Use it only in Push2Talk or Tap2Talk mode. 
/// Start speaking (press-and-hold mode).
public func startSpeech()

4sendAudioData

Send audio data to the server. 
/// Send audio data.
/// - Parameters:
///   - data: Pointer to audio data
///   - length: Data length
/// - Returns: Status code
public func sendAudioData(data: UnsafeMutablePointer<UInt8>, length: Int32) -> Int32?

5stopSpeech

Notify the server to stop uploading audio. Call this only in Push2Talk mode. 
/// Stop speaking (press-and-hold mode).
public func stopSpeech()

6 interrupt

Notify the server that the client wants to interrupt the current interaction and start speaking. Returns RequestAccepted. 
// Interrupt the current dialog.
public func interrupt()

7sendLocalRespondingStarted

Notify the server that the client has started playing TTS audio. 
/// Send a signal that response playback has started.
public func sendLocalRespondingStarted()

8sendLocalRespondingEnded

Notify the server that the client has stopped playing TTS audio. 
/// Send a signal that response playback has ended.
public func sendLocalRespondingEnded()

9 stop

End the current round of dialog. Disconnect from the server. 
/// Disconnect.
public func stop()

10requestToRespond

Initiate text-to-speech (TTS) synthesis from the client side. Or send other requests—such as image requests—to the server. 
/// Request a response.
/// - Parameters:
///   - type: Request type
///   - text: Request text
///   - params: Additional parameters
public func requestToRespond(type:String, text:String, params:[String : Any])

11updateInfo

Update parameter information. 
/// Update information.
/// - Parameter params: Parameters to update
public func updateInfo(params:[String : Any])

Callback

Callbacks for your app layer. 
/// Callback when the server connection succeeds.
public var onConnected: (() -> Void)?

/// Callback when the dialog protocol completes and you can start speaking.
public var onConversationStarted: (() -> Void)?

/// Callback for audio events: SpeechStarted, SpeechEnded, RespondingStarted, RespondingEnded.
public var onConversationEvent: ((DialogEvent) -> Void)?

/// Callback when the dialog state changes: Idle, Listening, Thinking, or Responding.
public var onConversationStatechanged: ((DialogState) -> Void)?

/// Callback when speech recognition text is received.
public var onSpeechContent: (([String: Any]?) -> Void)?

/// Callback when the LLM response text is received.
public var onRespondingContent: (([String: Any]?) -> Void)?

/// Callback for errors.
public var onErrorReceived: ((TYError) -> Void)?

/// Callback for synthesized audio data.
public var onSynthesizedData:  ((UnsafeMutablePointer<UInt8>, Int32) -> Void)?

/// Callback when the WebSocket connection closes.
public var onWebsocketClosed:((_ code: Int,_ reason: String?) -> Void)?

MultiModalRequestParam

Request parameter class All request parameters support builder-style setup. See the table below for parameter values and descriptions. These are parameters your client must configure or may optionally configure.

Parameters for the Start Connection Request

Top-level parameter

Second-level parameter

Third-level parameter

Fourth-level parameter

Type

Required

Description

input

workspace_id

string

Yes

Your workspace ID

app_id

string

Yes

Your App ID, created in the console. Use it to identify which dialog system to use.

dialog_id

string

No

Dialog ID. If set, continue the existing conversation.

parameters

upstream

type

string

Yes

Uplink type:

AudioOnly: Audio-only calls

AudioAndVideo: Upload video

mode

string

No

Client mode. Options:

  • push2talk

  • tap2talk

  • duplex

Default: tap2talk

audio_format

string

No

Audio format. Supports PCM and Opus. Default: PCM.

downstream

voice

string

No

Timbre of synthetic speech

sample_rate

int

No

Sample rate for speech synthesis. Default: 24000 Hz.

intermediate_text

string

No

Which intermediate text to return:

transcript: Return speech recognition results

dialog: Return intermediate responses from the dialog system

Use comma-separated values. Default: transcript

audio_format

string

No

Audio format. Supports PCM and MP3. Default: PCM.

client_info

user_id

string

Yes

End user ID. Used for user-specific processing.

device

uuid

string

No

A globally unique client ID. Generate it yourself and pass it to the SDK.

network

ip

string

No

Public IP address of the caller.

location

latitude

string

No

Caller Dimensions

longitude

string

No

Longitude of the caller.

city_name

string

No

City where the caller is located.

biz_params

user_defined_params

object

No

Other parameters to pass through to the agent.

user_defined_tokens

object

No

Authentication tokens to pass through to the agent.

tool_prompts

object

No

Prompts to pass through to the agent.

Parameters for requestToRespond

Top-level parameter

Second-level parameter

Third-level parameter

Type

Required

Description

input

type

string

Yes

Interaction type the service should use:

transcript: Convert text directly to speech

prompt: Send text to the large language model for a response

text

string

Yes

Text to process. Can be an empty string (""). Must not be null.

parameters

images

list[]

No

Images to analyze.

biz_params

object

No

Same as biz_params in the Start request. Passes custom parameters to the dialog system. This biz_params applies only to the current request.

Parameters for updateInfo

Top-level parameter

Second-level parameter

Third-level parameter

Type

Required

Description

parameters

images

list[]

No

Image data.

client_info

status

object

No

Current client status.

biz_params

object

No

Same as biz_params in the Start request. Passes custom parameters to the dialog system.

Dialog States (DialogState)

The voice chat service has three states: LISTENING, THINKING, and RESPONDING. 
IDLE("Idle"), System has not connected yet.
LISTENING("Listening"), Bot is listening for user input. You can send audio.
THINKING("Thinking"), Bot is thinking.
RESPONDING("Responding"), Bot is generating or playing back speech.

LLM Output from Dialog

Dialog results are delivered through the onMessageReceived callback. The format is shown below.

Top-level parameter

Second-level parameter

Third-level parameter

Type

Required

Description

output

event

string

Yes

Event name. Example: RequestAccepted

dialog_id

string

Yes

Dialog ID.

round_id

string

Yes

ID for this round of interaction.

llm_request_id

string

Yes

Request ID for the LLM call.

text

string

Yes

Text output by the system. Sent as a complete stream.

spoken

string

Yes

Text used for speech synthesis. Sent as a complete stream.

finished

bool

Yes

Whether output has finished.

extra_info

object

No

Additional info. Supported fields:

commands: Command string

agent_info: Agent info

tool_calls: Plugin response

dialog_debug: Dialog debug info

timestamps: Timestamps for each node in the link

Call Sequence Diagram

image

Usage Examples

Initialize Dialog Parameters

Use builder methods in the MultiModalRequestParam class to build parameters. 
var params = MultiModalRequestParam{ multiBuilder in
            multiBuilder.upStream = MultiModalRequestParam.UpStream(builder: { upstreamBuilder in
                upstreamBuilder.mode = DialogMode.duplex.rawValue
                upstreamBuilder.type = "AudioOnly"
            })
            multiBuilder.clientInfo = MultiModalRequestParam.ClientInfo(builder: {
                clientInfoBuilder in
                clientInfoBuilder.userId = "test-ios-user"
                clientInfoBuilder.device = MultiModalRequestParam.ClientInfo.Device(uuid: "12345")
            })
            multiBuilder.downStream = MultiModalRequestParam.DownStream(builder: {
                downStreamBuilder in
                downStreamBuilder.sampleRate = 48000
            })
        }

Create a MultiModalDialog Object 

self.conversation = MultiModalDialog(url: self.HOST,  workSpaceId:self.WORKSPACE_ID ,
     appId: self.APP_ID, mode: DialogMode.duplex)

Complete Usage Example

See the ChatViewController calls in the Demo project downloaded in the Environment and Dependencies section.

More SDK Interface Details

VQA Interaction

VQA lets users send images during a dialog to enable multimodal interaction with both images and voice. The core flow starts with a voice or text request that triggers a photo-capture intent. That intent sends the "visual_qa" command. After receiving the photo-capture command, send an image URL or Base64-encoded image data. Images must be under 180 KB. Send the photo-capture request after connecting. 
// Create image request parameters.
private func createImageParams() -> [String: Any]{
    var imageObject:[String: Any] = [
        "type":"url",
        "value":self.image_url ?? ""
    ]
    // Or upload a Base64-encoded image. Only images under 180 KB are supported.
    var imageObjectBase64:[String: Any] = [
                "type":"base64",
                "value":"imagebase64"
    ]
    var images = [imageObject]

    var updateParam = MultiModalRequestParam{ multiBuilder in
        multiBuilder.images = images
    }
    return updateParam.parameters
}
// Send the request.
self.conversation?.requestToRespond(type: "prompt", text: "", params: self.createImageParams())

Call LiveAI over WebSocket

LiveAI (video calling) is an official agent provided by Alibaba Cloud Model Studio for multimodal interaction. Using the iOS Lite SDK, you can call LiveAI over WebSocket by recording video frames yourself. Note: When calling LiveAI over WebSocket, only Base64-encoded images are supported. Each image must be under 180 KB.
  • LiveAI call sequence

Screenshot

  • Key code example
Follow the steps below to call LiveAI over WebSocket. 
// 1. Set interaction type to AudioAndVideo.
multiBuilder.upStream = MultiModalRequestParam.UpStream(builder: { upstreamBuilder in
                upstreamBuilder.mode = mode.rawValue
                upstreamBuilder.type = "AudioAndVideo"
            })
// 2. Initialize the MultiModalDialog instance. Choose RTC chain mode: ChainMode.RTC.
self.conversation = MultiModalDialog(url: self.HOST, chainMode: self.chain,  workSpaceId:self.WORKSPACE_ID ,
                                             appId: self.APP_ID, mode: mode)

// 3. After connecting, send a voicechat_video_channel request.
private func createVideoChatParams() -> [String: Any]{
        var video:[String: Any] = [
            "action":"connect",
            "type" : "voicechat_video_channel"
        ]
        var videos = [video]

        var updateParam = MultiModalRequestParam{ multiBuilder in
            multiBuilder.bizParams = MultiModalRequestParam.BizParams(builder: {
                bizBuilder in
                bizBuilder.videos = videos
            })
        }
        return updateParam.parameters
}

// 4. Submit one video frame every 500 ms. Make sure each image is under 180 KB.
/**
 * The demo does not call this. It shows how to use LiveAI over WebSocket.
 * Start streaming video frames. Send one frame every 500 ms.
 */
// Start sending images in the background.
func startPeriodicExecution() {
    timer = DispatchSource.makeTimerSource(queue: queue)
    timer?.schedule(deadline: .now(), repeating: .milliseconds(500))
    timer?.setEventHandler { [weak self] in
        self?.sendLocalImageBase64()
    }
    timer?.resume()
}

// Update image Base64.
private func sendLocalImageBase64() {
    print("send_local_image_base64() called : \(Thread.current)")
    var imageObjectBase64:[String: Any] = [
                "type":"base64",
                "value":getImageAsBase64FromAssets(imageName:"bridge") // Encode image to Base64.
    ]
    var images = [imageObjectBase64]

    var updateParam = MultiModalRequestParam{ multiBuilder in
        multiBuilder.images = images
    }

    self.conversation?.updateInfo(params: updateParam.parameters)

}
// See the full example in the demo code.

Text-to-Speech (TTS) 

self.conversation?.requestToRespond(type: "prompt", text: "Happiness is a skill—the inner peace you gain after letting go of unnecessary external desires.", params:[:])

Custom Prompt Variables and Values

  • In the console, go to your project's Prompt settings. Add a custom variable.
For example, the figure below defines a user_name field for the user's nickname. It inserts the variable user_name into the prompt as a placeholder: ${user_name}.
image.png
  • Set the variable in code.
In the example below, set "user_name" = "Da Mi". 
// Pass biz_params.user_prompt_params in the request parameter builder.
var promptParam = ["user_name" : "Da Mi"]
var params = MultiModalRequestParam{ multiBuilder in
            multiBuilder.bizParams = MultiModalRequestParam.BizParams(builder: {
                bizBuilder in
                bizBuilder.userPromptParams = promptParam
            })
}
  • Request response
image.png
Agent Development
Agent Context
Agent Governance & Evolution
Store
Application Marketplace
Use Cases
Support