Skip to main content

Sambert speech synthesis Java SDK

This topic describes the parameters and interface details of the Sambert speech synthesis Java software development kit (SDK).

User guide: For more information about model introductions and selection suggestions, see Real-time Speech Synthesis - CosyVoice/Sambert. Online experience: Not supported.

Service address

You must set the SDK service endpoint to the following address (which includes WorkspaceId) before initialization.
Sambert is only available in the China (Beijing) region.
Alibaba Cloud Model Studio has released a workspace-specific domain for the China (Beijing) region. The new dedicated domain delivers superior performance and higher stability for inference requests. We recommend migrating from dashscope.aliyuncs.com to {WorkspaceId}.cn-beijing.maas.aliyuncs.com.Replace {WorkspaceId} with your actual Workspace ID. The existing domain remains fully functional.
Service endpoint: wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference Replace {WorkspaceId} with your actual Workspace ID.

Prerequisites

You have activated the service and obtained and configured an API key. To prevent security risks from code leaks, set the API key as an environment variable instead of hard-coding it in your code.

Getting started

The SpeechSynthesizer class provides interfaces for non-streaming and one-way streaming calls. You can select a call method based on your business scenario:
  • Non-streaming call: After you submit the text, the server immediately processes it and returns the complete speech synthesis result. The entire process is blocking. The client must wait for the server to finish processing before proceeding to the next operation. This method is suitable for short text synthesis scenarios.
  • One-way streaming call: You can send the text to the server at one time and receive the speech synthesis results in real time. You cannot send the text in segments. This method is suitable for scenarios that require high real-time performance.

Non-streaming call

You can submit a single speech synthesis task to obtain the complete result at once. This method does not require a callback and does not stream intermediate results.
image
Instantiate the SpeechSynthesizer class. Call the call method to attach the request parameters. Then, you can perform synthesis and retrieve the binary audio data.
The following example shows how to use the non-streaming interface to call the Zhichu (sambert-zhichu-v1) speaker model to synthesize the text "What's the weather like today?" into audio with a sample rate of 48 kHz in the WAV audio format and save it to a file named output.wav.
import com.alibaba.dashscope.audio.tts.SpeechSynthesizer;
import com.alibaba.dashscope.audio.tts.SpeechSynthesisParam;
import com.alibaba.dashscope.audio.tts.SpeechSynthesisAudioFormat;
import com.alibaba.dashscope.utils.Constants;

import java.io.*;
import java.nio.ByteBuffer;

public class Main {
    public static void syncAudioDataToFile() {
        SpeechSynthesizer synthesizer = new SpeechSynthesizer();
        SpeechSynthesisParam param = SpeechSynthesisParam.builder()
                // If you do not set the API key as an environment variable, uncomment the following line and replace yourApiKey with your actual API key.
                // .apiKey("yourApiKey")
                .model("sambert-zhichu-v1")
                .text("What's the weather like today?")
                .sampleRate(48000)
                .format(SpeechSynthesisAudioFormat.WAV)
                .build();

        File file = new File("output.wav");
        // Submit a non-streaming synthesis task to get the complete audio data.
        ByteBuffer audio = synthesizer.call(param);
        try (FileOutputStream fos = new FileOutputStream(file)) {
            fos.write(audio.array());
            System.out.println("Synthesis done!");
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    public static void main(String[] args) {
        // Replace "{WorkspaceId}" with your actual Workspace ID.
        Constants.baseWebsocketApiUrl = "wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference";
        syncAudioDataToFile();
        System.exit(0);
    }
}

One-way streaming call

You can submit a single speech synthesis task and stream the intermediate results through a callback. The synthesis results are streamed through the callback method in ResultCallback.
image
Instantiate the SpeechSynthesizer class. Call the call method to attach the request parameters and the callback interface (ResultCallback) to start speech synthesis. You can retrieve the synthesis results in real time through the onEvent method of the callback interface (ResultCallback). After speech synthesis is complete (after the onComplete method of the ResultCallback interface is called), you can also call the getAudioData and getTimestamps methods of the SpeechSynthesizer class to retrieve the complete audio and timestamp results at once.
The following example shows how to use the streaming interface to call the Zhichu (sambert-zhichu-v1) speaker model to synthesize the text "What's the weather like today?" into streaming audio with a sample rate of 48 kHz and the default WAV audio format, and obtain the corresponding timestamps.
import com.alibaba.dashscope.audio.tts.SpeechSynthesisParam;
import com.alibaba.dashscope.audio.tts.SpeechSynthesisResult;
import com.alibaba.dashscope.audio.tts.SpeechSynthesizer;
import com.alibaba.dashscope.common.ResultCallback;
import com.alibaba.dashscope.utils.Constants;

import java.util.concurrent.CountDownLatch;

public class Main {
    public static void main(String[] args) {
        // Replace "{WorkspaceId}" with your actual Workspace ID.
        Constants.baseWebsocketApiUrl = "wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference";
        CountDownLatch latch = new CountDownLatch(1);
        SpeechSynthesizer synthesizer = new SpeechSynthesizer();
        SpeechSynthesisParam param = SpeechSynthesisParam.builder()
                // If you do not set the API key as an environment variable, uncomment the following line and replace yourApiKey with your actual API key.
                // .apiKey("yourApiKey")
                .model("sambert-zhichu-v1")
                .text("What's the weather like today?")
                .sampleRate(48000)
                .enableWordTimestamp(true)
                .enablePhonemeTimestamp(true)
                .build();

        class ReactCallback extends ResultCallback<SpeechSynthesisResult> {
            @Override
            public void onEvent(SpeechSynthesisResult result) {
                if (result.getAudioFrame() != null) {
                    // do something with the audio frame
                    System.out.println("Audio result length: " + result.getAudioFrame().array().length);
                }
                if (result.getTimestamp() != null) {
                    // do something with the timestamp
                    System.out.println("Timestamp: " + result.getTimestamp());
                }
            }

            @Override
            public void onComplete() {
                // do something when the synthesis is done
                System.out.println("onComplete!");
                latch.countDown();
            }

            @Override
            public void onError(Exception e) {
                // do something when an error occurs
                System.out.println("onError:" + e);
                latch.countDown();
            }
        }

        synthesizer.call(param, new ReactCallback());
        try {
            latch.await();
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.exit(0);
    }
}

Call through Flowable

Flowable is an open source framework for workflow and business process management (BPM). It is released under the Apache 2.0 license. For more information, see the Flowable API details.
The following example shows how to use the blockingForEach interface of the Flowable object to obtain the audio data and timestamp information (SpeechSynthesisResult) that is returned in each stream in a blocking manner.After all streaming data from Flowable is returned, you can also call the getAudioData and getTimestamps methods of the SpeechSynthesizer class to retrieve the complete synthesis result and timestamps.
import com.alibaba.dashscope.audio.tts.SpeechSynthesisParam;
import com.alibaba.dashscope.audio.tts.SpeechSynthesisResult;
import com.alibaba.dashscope.audio.tts.SpeechSynthesizer;
import com.alibaba.dashscope.utils.Constants;
import io.reactivex.Flowable;

public class Main {
    public static void main(String[] args) {
        // Replace "{WorkspaceId}" with your actual Workspace ID.
        Constants.baseWebsocketApiUrl = "wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference";
        SpeechSynthesizer synthesizer = new SpeechSynthesizer();
        SpeechSynthesisParam param = SpeechSynthesisParam.builder()
                // If you do not set the API key as an environment variable, uncomment the following line and replace yourApiKey with your actual API key.
                // .apiKey("yourApiKey")
                .model("sambert-zhichu-v1")
                .text("What's the weather like today?")
                .sampleRate(48000)
                .enableWordTimestamp(true)
                .build();

        Flowable<SpeechSynthesisResult> flowable = synthesizer.streamCall(param);
        flowable.blockingForEach(
                msg -> {
                    if (msg.getAudioFrame() != null) {
                        // do something with the audio frame
                        System.out.println("getAudioFrame");
                    }
                    if (msg.getTimestamp() != null) {
                        // do something with the timestamp
                        System.out.println("getTimestamp");
                    }
                }
        );
        System.exit(0);
    }
}

High-concurrency calls

The DashScope Java SDK uses the connection pool technology of OkHttp3 to reduce the overhead of repeatedly establishing connections. For more information, see High-concurrency scenarios.

Request parameters

You can use the chained method of SpeechSynthesisParam to configure parameters, such as the model and the text to be synthesized. The configured object is passed to the call method of the SpeechSynthesizer class.
SpeechSynthesisParam param = SpeechSynthesisParam.builder()
                .model("sambert-zhichu-v1")
                .text("What's the weather like today?")
                .sampleRate(48000)
                .enableWordTimestamp(true)
                .build();
ParameterTypeDefault valueRequiredDescription
modelString
YesThe name of the timbre model used for speech synthesis. For a complete list, see Model list.
textString
YesThe text to be synthesized. The text must be UTF-8 encoded and cannot be empty.Maximum number of characters: 10,000.
Character calculation rule: One Chinese character, one English letter, one punctuation mark, or one space between sentences is counted as one character.
The Speech Synthesis Markup Language (SSML) format is supported. For more information about how to use SSML, see SSML overview.
formatenumWAVNoThe encoding format of the synthesized audio. The following formats are supported:
  • SpeechSynthesisAudioFormat.PCM
  • SpeechSynthesisAudioFormat.WAV
  • SpeechSynthesisAudioFormat.MP3
Import SpeechSynthesisAudioFormat using import com.alibaba.dashscope.audio.tts.SpeechSynthesisAudioFormat;.
sampleRateint16000NoThe sample rate of the synthesized audio in Hz. We recommend that you use the default sample rate of the model. For more information, see Model list. If the sample rate does not match, the service performs necessary upsampling or downsampling.
volumeint50NoThe volume of the synthesized audio. Valid values: 0 to 100.
ratefloat1.0NoThe speech rate of the synthesized audio. Valid values: 0.5 to 2.
  • 0.5: indicates 0.5 times the default speech rate.
  • 1: indicates the default speech rate. The default speech rate is the default output speech rate of the model. The speech rate may vary slightly depending on the speaker. The default speech rate is about four characters per second.
  • 2: indicates twice the default speech rate.
pitchfloat1.0NoThe pitch of the synthesized audio. Valid values: 0.5 to 2.
enableWordTimestampbooleanfalseNoSpecifies whether to enable word-level timestamps. The default value is false.
enablePhonemeTimestampbooleanfalseNoSpecifies whether to display phoneme-level timestamps when word-level timestamps are enabled (enableWordTimestamp is true). The default value is false.
apiKeyString
NoThe user's API key.

Key interfaces

SpeechSynthesizer class

You can import SpeechSynthesizer using import com.alibaba.dashscope.audio.tts.SpeechSynthesizer;. The key interfaces are as follows:
Interface/MethodParameterReturn valueDescription
public ByteBuffer call(SpeechSynthesisParam param)
param: request parametersBinary audioSends the text to be synthesized and obtains the speech synthesis result. This method blocks the current thread until all results are returned.
public void call(SpeechSynthesisParam param, ResultCallback<SpeechSynthesisResult> callback)
NoneAsynchronously starts a speech synthesis task.After the task is started, the server calls the method of the ResultCallback instance through a callback to return key process information and data to the client.
public ByteBuffer getAudioData()
NoneBinary audioObtains the complete binary audio data.When you use a one-way streaming call, call this method to get the complete audio at once after the callback is complete (after the onComplete method of ResultCallback is called).
public List<Sentence> getTimestamps()
NoneList collection of sentence-level timestamp information (Sentence)Obtains the complete sentence-level timestamp information (Sentence).When you use a one-way streaming call, call this method to get the complete timestamp at once after the callback is complete (after the onComplete method of ResultCallback is called).
public String getLastRequestId()
NoneRequest ID of the current taskObtains the request ID of the current task. You can call this method after you call call to start a new task.
public long getFirstPackageDelay()
NoneFirst package latency of the current taskObtains the first package latency of the current task. Use this method after the task is complete.

Callback interface (ResultCallback)

When you use a one-way streaming call, you can retrieve the synthesis result through the ResultCallback interface.
ResultCallback<SpeechSynthesisResult> callback = new ResultCallback<SpeechSynthesisResult>() {
    @Override
    public void onEvent(SpeechSynthesisResult result) {
        System.out.println("Request ID: " + result.getRequestId());
        // Implement the logic to process the speech synthesis result here.
    }

    @Override
    public void onComplete() {
        System.out.println("Task complete");
    }

    @Override
    public void onError(Exception e) {
        System.out.println("Task failed: " + e.getMessage());
    }
};
Interface/MethodParameterReturn valueDescription
public void onEvent(SpeechSynthesisResult result)
result: audio data and timestamp information (SpeechSynthesisResult)NoneThis method is called back when the server returns synthetic data.
public void onComplete()
NoneNoneThis method is called back after all synthetic data is returned.
public void onError(Exception e)
e: exception informationNoneThis method is called back when an exception occurs during the call or the service returns an error.

Response results

Non-streaming call: The response is binary audio data. One-way streaming call: The response is audio data and timestamp information (SpeechSynthesisResult).

Audio data and timestamp information (SpeechSynthesisResult)

SpeechSynthesisResult encapsulates the speech synthesis result. The commonly used interfaces are getAudioFrame and getTimestamp.
Interface/MethodParameterReturn valueDescription
public ByteBuffer getAudioFrame()
NoneBinary audio dataReturns the incremental binary audio data of a streaming synthesis segment. The value can be empty.
public List<Sentence> getTimestamp()
NoneList collection of sentence-level timestamp information (Sentence)Obtains sentence-level timestamp information in batches. The value can be empty.

Sentence-level timestamp information (Sentence)

Sentence encapsulates sentence-level timestamp information.
Interface/MethodParameterReturn valueDescription
public int getBeginTime()
NoneStart time of the sentence in msReturns the start time of the sentence.
public int getEndTime()
NoneEnd time of the sentence in msReturns the end time of the sentence.
public List<Word> getWords()
NoneList collection of word-level timestamp information (Word)Obtains word-level timestamp information in batches. The value can be empty.

Word-level timestamp information (Word)

Word encapsulates word-level timestamp information.
Interface/MethodParameterReturn valueDescription
public int getBeginTime()
NoneStart time of the word in msReturns the start time of the word.
public int getEndTime()
NoneEnd time of the word in msReturns the end time of the word.
public String getText()
NoneText informationReturns the text information.
public List<Phoneme> getPhonemes()
NoneList collection of phoneme-level timestamp information (Phoneme)Obtains phoneme-level timestamp information in batches. The value can be empty.

Phoneme-level timestamp information (Phoneme)

Phoneme encapsulates phoneme-level timestamp information.
Interface/MethodParameterReturn valueDescription
public int getBeginTime()
NoneStart time of the phoneme in msReturns the start time of the phoneme.
public int getEndTime()
NoneEnd time of the phoneme in msReturns the end time of the phoneme.
public String getText()
NoneText informationReturns the text information.
public String getTone()
NoneToneReturns the tone.
  • In English, 0, 1, and 2 represent unstressed, primary stress, and secondary stress, respectively.
  • In Pinyin, 1, 2, 3, 4, and 5 represent the first, second, third, fourth, and neutral tones, respectively.

Error codes

If an API call fails and returns an error message, see Error codes to resolve the issue.

More examples

For more examples, see GitHub.

FAQ

For more information, see the QA on GitHub.

Model list

The default sample rate is the optimal sample rate for the current model. By default, the output is based on this sample rate. Downsampling and upsampling are also supported. For example, for the Zhimiao voice, the default sample rate is 16 kHz. You can downsample it to 8 kHz, but upsampling it to 48 kHz does not provide additional benefits.
TimbreAudio sample (Right-click to save the audio)model parameterTimestamp supportScenariosFeaturesLanguageDefault sample rate (Hz)
Zhinansambert-zhinan-v1YesGeneral scenariosAdvertising male voiceChinese and English48 kHz
Zhiqisambert-zhiqi-v1YesGeneral scenariosGentle female voiceChinese and English48 kHz
Zhichusambert-zhichu-v1YesNews broadcastingA Bite of China male voiceChinese and English48 kHz
Zhidesambert-zhide-v1YesNews broadcastingNews male voiceChinese and English48 kHz
Zhijiasambert-zhijia-v1YesNews broadcastingStandard female voiceChinese and English48 kHz
Zhirusambert-zhiru-v1YesNews broadcastingNews female voiceChinese and English48 kHz
Zhiqiansambert-zhiqian-v1YesDubbing and news broadcastingInformation female voiceChinese and English48 kHz
Zhixiangsambert-zhixiang-v1YesDubbingMagnetic male voiceChinese and English48 kHz
Zhiweisambert-zhiwei-v1YesReading product introductionsLolita female voiceChinese and English48 kHz
Zhihaosambert-zhihao-v1YesGeneral scenariosConsultation male voiceChinese and English16 KB
Zhijingsambert-zhijing-v1YesGeneral scenariosStrict female voiceChinese and English16 k
Zhimingsambert-zhiming-v1YesGeneral scenariosHumorous male voiceChinese and English16 KB
Zhimosambert-zhimo-v1YesGeneral scenariosEmotional male voiceChinese and English16 kHz
Zhinasambert-zhina-v1YesGeneral scenariosZhejiang Mandarin female voiceChinese and English16 kHz
Zhishusambert-zhishu-v1YesGeneral scenariosInformation male voiceChinese and English16 KB
Zhishasambert-zhistella-v1YesGeneral scenariosIntellectual female voiceChinese and English16 kHz
Zhitingsambert-zhiting-v1YesGeneral scenariosRadio female voiceChinese and English16 kHz
Zhixiaosambert-zhixiao-v1YesGeneral scenariosInformation female voiceChinese and English16 kHz
Zhiyasambert-zhiya-v1YesGeneral scenariosStrict female voiceChinese and English16 kHz
Zhiyesambert-zhiye-v1YesGeneral scenariosYoung male voiceChinese and English16 kHz
Zhiyingsambert-zhiying-v1YesGeneral scenariosCute child voiceChinese and English16 kHz
Zhiyuansambert-zhiyuan-v1YesGeneral scenariosCaring sister voiceChinese and English16 kHz
Zhiyuesambert-zhiyue-v1YesCustomer serviceGentle female voiceChinese and English16 kHz
Zhiguisambert-zhigui-v1YesReading product introductionsLivestreaming female voiceChinese and English16 kHz
Zhishuosambert-zhishuo-v1YesDigital humanNatural male voiceChinese and English16 kHz
Zhimiao (multi-emotional)sambert-zhimiao-emo-v1YesReading product introductions, digital humans, and livestreamingMulti-emotional female voiceChinese and English16 kHz
Zhimaosambert-zhimao-v1YesReading product introductions, dubbing, digital humans, and livestreamingLivestreaming female voiceChinese and English16 kHz
Zhilunsambert-zhilun-v1YesDubbingSuspense narrationChinese and English16 kHz
Zhifeisambert-zhifei-v1YesDubbingPassionate narrationChinese and English16 KB
Zhidasambert-zhida-v1YesNews broadcastingStandard male voiceChinese and English16 KB
Camilasambert-camila-v1NoGeneral scenariosSpanish female voiceSpanish16 kHz
Perlasambert-perla-v1NoGeneral scenariosItalian female voiceItalian16 k
Indahsambert-indah-v1NoGeneral scenariosIndonesian female voiceIndonesian16 kHz
Clarasambert-clara-v1NoGeneral scenariosFrench female voiceFrench16 kHz
Hannasambert-hanna-v1NoGeneral scenariosGerman female voiceGerman16 kHz
Bethsambert-beth-v1YesGeneral scenariosConsultation female voiceAmerican English16 kHz
Bettysambert-betty-v1YesGeneral scenariosCustomer service female voiceAmerican English16 kHz
Callysambert-cally-v1YesGeneral scenariosNatural female voiceAmerican English16 kHz
Cindysambert-cindy-v1YesGeneral scenariosConversational female voiceAmerican English16 kHz
Evasambert-eva-v1YesGeneral scenariosCompanion female voiceAmerican English16 KB
Donnasambert-donna-v1YesGeneral scenariosEducation female voiceAmerican English16 kHz
Briansambert-brian-v1YesGeneral scenariosCustomer service male voiceAmerican English16 kHz
Waansambert-waan-v1NoGeneral scenariosThai female voiceThai16 kHz
Text Generation
Image Generation
Video Generation
World models
3D Model Generation
Audio
  • Audio generation
Realtime API
Text Embedding
Decision Model
TokenPlan
Model Production