Skip to main content

Sambert speech synthesis Python SDK

This topic describes the parameters and interfaces of the Sambert speech synthesis Python SDK.

Alibaba Cloud Model Studio has released a workspace-specific domain for the China (Beijing) region. The new dedicated domain delivers superior performance and higher stability for inference requests. We recommend migrating from dashscope.aliyuncs.com to {WorkspaceId}.cn-beijing.maas.aliyuncs.com.Replace {WorkspaceId} with your actual Workspace ID. The existing domain remains fully functional.
User guide: For more information about model introductions and selection suggestions, see Real-time Speech Synthesis - CosyVoice/Sambert. Online experience: Not supported.

Prerequisites

You have activated the service and obtained and configured an API key. To prevent security risks from code leaks, set the API key as an environment variable instead of hard-coding it in your code.

Model list

The default sample rate is the optimal sample rate for the current model. By default, the output is based on this sample rate. Downsampling and upsampling are also supported. For example, for the Zhimiao voice, the default sample rate is 16 kHz. You can downsample it to 8 kHz, but upsampling it to 48 kHz does not provide additional benefits.
TimbreAudio sample (Right-click to save the audio)model parameterTimestamp supportScenariosFeaturesLanguageDefault sample rate (Hz)
Zhinansambert-zhinan-v1YesGeneral scenariosAdvertising male voiceChinese and English48 kHz
Zhiqisambert-zhiqi-v1YesGeneral scenariosGentle female voiceChinese and English48 kHz
Zhichusambert-zhichu-v1YesNews broadcastingA Bite of China male voiceChinese and English48 kHz
Zhidesambert-zhide-v1YesNews broadcastingNews male voiceChinese and English48 kHz
Zhijiasambert-zhijia-v1YesNews broadcastingStandard female voiceChinese and English48 kHz
Zhirusambert-zhiru-v1YesNews broadcastingNews female voiceChinese and English48 kHz
Zhiqiansambert-zhiqian-v1YesDubbing and news broadcastingInformation female voiceChinese and English48 kHz
Zhixiangsambert-zhixiang-v1YesDubbingMagnetic male voiceChinese and English48 kHz
Zhiweisambert-zhiwei-v1YesReading product introductionsLolita female voiceChinese and English48 kHz
Zhihaosambert-zhihao-v1YesGeneral scenariosConsultation male voiceChinese and English16 KB
Zhijingsambert-zhijing-v1YesGeneral scenariosStrict female voiceChinese and English16 k
Zhimingsambert-zhiming-v1YesGeneral scenariosHumorous male voiceChinese and English16 KB
Zhimosambert-zhimo-v1YesGeneral scenariosEmotional male voiceChinese and English16 kHz
Zhinasambert-zhina-v1YesGeneral scenariosZhejiang Mandarin female voiceChinese and English16 kHz
Zhishusambert-zhishu-v1YesGeneral scenariosInformation male voiceChinese and English16 KB
Zhishasambert-zhistella-v1YesGeneral scenariosIntellectual female voiceChinese and English16 kHz
Zhitingsambert-zhiting-v1YesGeneral scenariosRadio female voiceChinese and English16 kHz
Zhixiaosambert-zhixiao-v1YesGeneral scenariosInformation female voiceChinese and English16 kHz
Zhiyasambert-zhiya-v1YesGeneral scenariosStrict female voiceChinese and English16 kHz
Zhiyesambert-zhiye-v1YesGeneral scenariosYoung male voiceChinese and English16 kHz
Zhiyingsambert-zhiying-v1YesGeneral scenariosCute child voiceChinese and English16 kHz
Zhiyuansambert-zhiyuan-v1YesGeneral scenariosCaring sister voiceChinese and English16 kHz
Zhiyuesambert-zhiyue-v1YesCustomer serviceGentle female voiceChinese and English16 kHz
Zhiguisambert-zhigui-v1YesReading product introductionsLivestreaming female voiceChinese and English16 kHz
Zhishuosambert-zhishuo-v1YesDigital humanNatural male voiceChinese and English16 kHz
Zhimiao (multi-emotional)sambert-zhimiao-emo-v1YesReading product introductions, digital humans, and livestreamingMulti-emotional female voiceChinese and English16 kHz
Zhimaosambert-zhimao-v1YesReading product introductions, dubbing, digital humans, and livestreamingLivestreaming female voiceChinese and English16 kHz
Zhilunsambert-zhilun-v1YesDubbingSuspense narrationChinese and English16 kHz
Zhifeisambert-zhifei-v1YesDubbingPassionate narrationChinese and English16 KB
Zhidasambert-zhida-v1YesNews broadcastingStandard male voiceChinese and English16 KB
Camilasambert-camila-v1NoGeneral scenariosSpanish female voiceSpanish16 kHz
Perlasambert-perla-v1NoGeneral scenariosItalian female voiceItalian16 k
Indahsambert-indah-v1NoGeneral scenariosIndonesian female voiceIndonesian16 kHz
Clarasambert-clara-v1NoGeneral scenariosFrench female voiceFrench16 kHz
Hannasambert-hanna-v1NoGeneral scenariosGerman female voiceGerman16 kHz
Bethsambert-beth-v1YesGeneral scenariosConsultation female voiceAmerican English16 kHz
Bettysambert-betty-v1YesGeneral scenariosCustomer service female voiceAmerican English16 kHz
Callysambert-cally-v1YesGeneral scenariosNatural female voiceAmerican English16 kHz
Cindysambert-cindy-v1YesGeneral scenariosConversational female voiceAmerican English16 kHz
Evasambert-eva-v1YesGeneral scenariosCompanion female voiceAmerican English16 KB
Donnasambert-donna-v1YesGeneral scenariosEducation female voiceAmerican English16 kHz
Briansambert-brian-v1YesGeneral scenariosCustomer service male voiceAmerican English16 kHz
Waansambert-waan-v1NoGeneral scenariosThai female voiceThai16 kHz

Quick start

The core resource (client.audio.tts) provides interfaces for non-streaming calls and one-way streaming calls. Choose the appropriate method based on your scenario:
  • Non-streaming call: Submit the text. The server processes it and returns the complete speech synthesis result. The entire process is blocking, which means the client must wait for the server to finish before it can proceed. This method is suitable for short text synthesis.
  • One-way streaming call: Send the text to the server in a single request and receive the speech synthesis result in real time. You cannot send the text in segments. This method is suitable for scenarios that require low latency.

Non-streaming call

Submit a single speech synthesis task for speech synthesis (intermediate results are not streamed) and retrieve the complete result at once.
image
Call the synthesize method of the core resource (client.audio.tts) to synthesize speech. You can use the synthesize method to set request parameters. After the task is complete, the method returns the complete audio data (bytes). Timestamp information is available only through a streaming session. For more information, see Response results.
The following example shows how to use the synchronous interface to call the Zhichu (sambert-zhichu-v1) model to synthesize the text "How is the weather today?". The synthesized audio is in the WAV format with a sample rate of 48 kHz and is saved to a file named output.wav.
# coding=utf-8
import os
import sys

import dashscope

# If you do not configure the API key as an environment variable, replace the following line with your API key: api_key="sk-xxx"
# The following configuration is for the China (Beijing) region. Replace "{WorkspaceId}" with your actual workspace ID when you call the API. The configuration varies by region.
client = dashscope.Client(
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    workspace="{WorkspaceId}",
    region="cn-beijing",
)
audio = client.audio.tts.synthesize(
    'How is the weather today?',
    model='sambert-zhichu-v1',
    sample_rate=48000,
    format='wav')
with open('output.wav', 'wb') as f:
    f.write(audio)
print('SUCCESS: get audio data: %dbytes in output.wav' % (sys.getsizeof(audio)))

One-way streaming call

Submit a single speech synthesis task. Intermediate results are streamed back, and the synthesis results are retrieved by iterating events()/audio_chunks() of the streaming synthesis session.
image
  1. Call the connect method of the core resource (client.audio.tts) to create a streaming synthesis session. The connection is automatically established when the with context is entered.
  2. Call the send_text method of the session to send the text to be synthesized, and call the stop method to notify the server that the input is complete.
  3. In the receiving thread, iterate events() (audio bytes and JSON events, including timestamps) or audio_chunks() (audio only) of the session to retrieve the synthesis results.
The following example shows how to use the streaming interface to call the Zhichu (sambert-zhichu-v1) model. It synthesizes the text "How is the weather today?" into streaming audio in the default WAV format with a 48 kHz sample rate, and retrieves the corresponding timestamp.
# coding=utf-8

import os
import sys

import dashscope

# If you do not configure the API key as an environment variable, replace the following line with your API key: api_key="sk-xxx"
# The following configuration is for the China (Beijing) region. Replace "{WorkspaceId}" with your actual workspace ID when you call the API. The configuration varies by region.
client = dashscope.Client(
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    workspace="{WorkspaceId}",
    region="cn-beijing",
)

# Create a streaming synthesis session: the connection is automatically established when the with context is entered
with client.audio.tts.connect(
        model='sambert-zhichu-v1',
        sample_rate=48000,
        parameters={'word_timestamp_enabled': True,
                    'phoneme_timestamp_enabled': True},
) as session:
    session.send_text('How is the weather today?')
    # Notify the server that the input is complete
    session.stop()
    # events() yields both audio bytes and server JSON events (including timestamps)
    for event in session.events():
        if isinstance(event, bytes):
            print('audio result length:', sys.getsizeof(event))
        else:
            output = (event.get('payload') or {}).get('output') or {}
            if output.get('timestamp') is not None:
                print('timestamp result:', str(output.get('timestamp')))

Request parameters

You can set the request parameters in the synthesize, stream, and connect methods of the core resource (client.audio.tts). Named parameters are passed directly, and word_timestamp_enabled and phoneme_timestamp_enabled are passed in the parameters dictionary.
ParameterTypeDefault valueRequiredDescription
modelstr
YesThe name of the voice model to use for speech synthesis. For a complete list, see Model list.
textstr
YesThe text to be synthesized. The text must be UTF-8 encoded and cannot be empty.Maximum number of characters: 10,000.
Character calculation rule: One Chinese character, one English letter, one punctuation mark, or one space between sentences is counted as one character.
The SSML format is supported. For more information, see Introduction to SSML.
formatstrwavNoThe encoding format of the synthesized audio. The pcm, wav, and mp3 formats are supported.
sample_rateint16000NoThe sample rate of the synthesized audio in Hz. We recommend that you use the model's default sample rate (see the Model list). If there is a mismatch, the service performs the necessary upsampling or downsampling.
volumeint50NoThe volume of the synthesized audio. Valid values: 0 to 100.
ratefloat1.0NoThe speech rate of the synthesized audio. Valid values: 0.5 to 2.
  • 0.5: indicates 0.5 times the default speech rate.
  • 1: indicates the default speech rate. The default speech rate is the synthesis rate of the default output of the model. The speech rate may vary slightly depending on the speaker. The rate is about four characters per second.
  • 2: indicates twice the default speech rate.
pitchfloat1.0NoThe pitch of the synthesized audio. Valid values: 0.5 to 2.
word_timestamp_enabledboolFalseNoSpecifies whether to enable word-level timestamps. The default value is false.
phoneme_timestamp_enabledboolFalseNoSpecifies whether to generate phoneme-level timestamp information based on word-level timestamps (word_timestamp_enabled). The default value is false.

API reference

Core resource (client.audio.tts)

The speech synthesis capability is mounted on the client.audio.tts resource: synthesize returns the complete audio at once, stream yields the audio chunks, and connect creates a streaming synthesis session. The key interfaces are as follows:
Interface/MethodParametersReturn valueDescription
def synthesize(self,
               text: str,
               *,
               model: str,
               voice: str = "longxiaochun",
               format: str = "mp3",
               sample_rate: int = 22050,
               volume: int = None,
               rate: float = None,
               pitch: float = None,
               parameters: Dict = None,
               ...) -> bytes
  • text: The text to be synthesized.
  • model: The name of the model. For more information, see Model list.
  • The other parameters are request parameters, such as format.
The complete audio data (bytes)Starts a speech synthesis task and returns the complete audio data after the synthesis is complete.You can use the synthesize method to set the request parameters.
def stream(self,
           text: str,
           *,
           model: str,
           ...) -> Iterator[bytes]
Same as synthesizeThe audio data yielded chunk by chunk (Iterator[bytes])Starts a speech synthesis task, sends the text in a single request, and yields the audio chunks one by one. It yields only audio and does not expose per-event JSON metadata. To obtain timestamps, use the connect session.
def connect(self,
            *,
            model: str,
            format: str = "mp3",
            sample_rate: int = 22050,
            parameters: Dict = None,
            ...) -> SynthesisSession
The SynthesisSession streaming synthesis sessionCreates a streaming synthesis session. The connection is automatically established when the with context is entered. Use it together with send_text, stop, and events()/audio_chunks(). For more information, see Event iteration.

Event iteration (events()/audio_chunks())

When you make a one-way streaming call, the server returns key process information and data to the client as events. In the receiving thread, iterate events() or audio_chunks() of the session to process the information and data returned by the server.
# events() yields both audio bytes and server JSON events (including timestamps)
for event in session.events():
    if isinstance(event, bytes):
        print('Received binary audio data:', event)
    else:
        output = (event.get('payload') or {}).get('output') or {}
        if output.get('timestamp') is not None:
            print('Received timestamp data:', str(output.get('timestamp')))

# Alternatively, iterate only the audio (the session still updates last_response and usage)
for chunk in session.audio_chunks():
    print('Received binary audio data:', chunk)
Interface/MethodParametersReturn valueDescription
def send_text(self, text: str) -> None
text: The text to be synthesizedNoneSends the text to be synthesized. Sending the text and receiving the results must be performed concurrently.
def stop(self) -> None
NoneNoneNotifies the server that the input is complete. Then, continue to consume events()/audio_chunks() until all the remaining audio is returned.
def events(self) -> Generator[Union[Dict, bytes], None, None]
NoneDict (JSON event) or bytes (audio)Yields items as the server returns the synthesis data: audio as bytes frames and metadata (such as timestamps) as JSON event dict objects. Only one reader of events() or audio_chunks() is allowed per session.
def audio_chunks(self) -> Iterator[bytes]
NonebytesIterates only the audio data. The session still updates its last_response and usage.
session.first_package_delay
NonefloatThe delay (in milliseconds) from sending the text to receiving the first audio data chunk. Read this property after the task is complete.

Response results

Synthesis result (audio data and timestamps)

synthesize returns the complete audio bytes. In streaming calls, you receive the audio bytes chunk by chunk through stream or the audio_chunks() method of the session. Timestamp information is retrieved from the JSON events yielded by events() of the session (payload.output.timestamp).
Interface/MethodParametersReturn valueDescription
audio = client.audio.tts.synthesize(...)
NoneThe complete binary audio dataIn non-streaming synthesis, obtains the complete binary audio data.
for chunk in session.audio_chunks():
NoneThe current segment of synthesized binary audio dataIn streaming synthesis, receives the synthesized audio frame data chunk by chunk.
output = (event.get('payload') or {}).get('output') or {}
output.get('timestamp')
NoneThe currently synthesized sentence's timestamp informationIn streaming synthesis, obtains the timestamp information of the currently synthesized sentence from the JSON events yielded by events().
timestamps = [
    ((e.get('payload') or {}).get('output') or {}).get('timestamp')
    for e in json_events if ...
]
NoneThe timestamp information of all sentencesIn streaming synthesis, accumulating the timestamps of the sentences yielded by events() gives the timestamp information of all sentences.

Timestamp information

In the JSON events yielded by events() of the session, payload.output.timestamp is the timestamp information of the currently synthesized sentence. Accumulating the timestamps of all sentences gives the timestamp information of all sentences. The following code provides an example of the timestamp information for a single sentence. words corresponds to word-level timestamp information, and phonemes corresponds to phoneme-level timestamp information:
{
    "begin_time":0,
    "end_time":1412,
    "words":[
        {
            "text":"jin",
            "begin_time":0,
            "end_time":200,
            "phonemes":[
                {
                    "begin_time":0,
                    "end_time":82,
                    "text":"j_c",
                    "tone":1
                },
                {
                    "begin_time":82,
                    "end_time":200,
                    "text":"in_c",
                    "tone":1
                }
            ]
        }
    ]
}
The parameters are as follows:

Parameter

Type

Description

begin_time

int

The start time of a sentence, word, or phoneme in milliseconds.

end_time

int

The end time of a sentence, word, or phoneme in milliseconds.

words

list

The word-level timestamp information. The word_timestamp_enabled parameter in the request must also be set to true.

text

str

The text information.

phonemes

list

The phoneme-level timestamp information. The phoneme_timestamp_enabled parameter in the request must also be set to true.

tone

str

The tone.

  • In English, 0, 1, and 2 represent unstressed, primary stress, and secondary stress, respectively.

  • In Pinyin, 1, 2, 3, 4, and 5 represent the first, second, third, fourth, and neutral tones, respectively.

More examples

For more examples, see GitHub.

Error codes

If an API call fails and returns an error message, see Error codes to resolve the issue.