Skip to main content

Dataset

Datasets (formerly Data Connections) are the unified entry point for Alibaba Cloud Model Studio RAG to manage external data sources. By creating a dataset, applications can securely access data in enterprise databases, document systems, and object storage, and query and reference the data in real time during conversations.

This page describes RAG datasets (formerly Data Connections). The new Connector for agent scenarios is now available, supporting more connectable systems and MCP-based invocation by agents. Migrate legacy data connections before September 30, 2026 by following Data connection migration.
Datasets are the unified entry point for RAG to manage external data sources. When you create a dataset, you select a dataset type based on the data source. Applications can then securely access data in enterprise databases, document systems, and object storage, and query and reference the data in real time during conversations.

Dataset types

Datasets fall into two main categories based on how data is stored and accessed — platform-hosted and stream processing — with three dataset types in total:

Format

Dataset type

Storage

Scenarios

Platform-hosted

File

Alibaba Cloud Model Studio or your own OSS

Upload and manage unstructured documents (PDF, Word, Markdown, etc.), with support for local upload and continuous synchronization from external sources such as Lark, DingTalk, SharePoint, Yuque, and OSS

Table

Alibaba Cloud Model Studio or your own OSS

Import and query structured tabular data, such as CSV and Excel.

Stream processing

Database

Data remains in the source database for real-time access.

Connect to MySQL, PostgreSQL, or PolarDB-X 2.0 databases and execute SQL queries (supported only for data sources imported with DMS).

Yuque, OSS, DingTalk, Lark, and SharePoint are not created as standalone dataset types. They serve only as import and synchronization sources for file datasets. For details, see Data synchronization rules below.

Prerequisites

Before you create a dataset, ensure you meet the following prerequisites:
  • Account permissions: You must use a root user or a RAM user with permissions to manage datasets. A root user must authorize a RAM user before they can use the dataset feature. For details, see Permission Management.
  • Data source preparation:
    • File/Table dataset: You must have documents or spreadsheets ready for upload, or have an existing OSS Bucket.
    • MySQL database: You must have a MySQL database instance (Alibaba Cloud RDS or self-managed). Ensure the instance is accessible via a public or private network.
    • PostgreSQL database: You must have a PostgreSQL database instance with its wal_level parameter set to logical.
    • PolarDB-X 2.0 database: You must have an Alibaba Cloud PolarDB-X 2.0 instance in a region that supports private network access. If you plan to import the data source using DMS, you must first register the PolarDB-X instance in DMS.
    • Yuque sync source: You must have a Yuque Knowledge Base and a Personal Access Token. Only the public version of Yuque is supported.
    • OSS sync source: You must have an OSS Bucket and have activated Alibaba Cloud Vector Search.

Create a dataset

  1. Go to the Dataset page and click Create in the upper-right corner.
  2. Enter a dataset name (up to 20 characters) and a description (up to 200 characters). The description guides how accurately agents call the dataset, so include the data content and purpose.
  3. Select a dataset type (File, Table, or Database), and then configure the settings for that type:
    • File
    • Table
    • MySQL
    • PostgreSQL
    • PolarDB-X 2.0
    A file dataset is used to manage unstructured documents (such as PDF and Word) and supports local upload and continuous synchronization from external sources such as Lark, DingTalk, SharePoint, Yuque, and OSS.
    1. Select File for the dataset type.
    2. Select a storage location:
      • Use Platform Storage: Data is stored in the storage space provided by Alibaba Cloud Model Studio. This option offers a limited-time free quota of up to 100,000 files and 1 TB of storage.
      • Use Your Own OSS Storage: Data is stored in your own OSS bucket. This option is suitable for large-scale data storage.
        • If you are using this option for the first time, follow the on-screen instructions to complete the authorization.
        • The target bucket must have the bailian-connector-access tag with the value ReadAndWrite to allow access from Alibaba Cloud Model Studio. Add tag
  4. Click Confirm to complete the creation.

Import data

  • Import files
  • Import tables
  • Import files from OSS
  • Import RDS MySQL data
  • Import self-hosted MySQL data
  • Import data from self-hosted PostgreSQL
Click Details on the file dataset card to open the file management page.
  1. In the Category pane on the left, select an existing category or click the + icon to create a new one.
    Alibaba Cloud Model Studio manages imported files by category.
    Each workspace supports a maximum of 500 categories. To request a higher category limit, submit a ticket.
  2. Click Import Data to open the Import Data page. For the import method, select Upload Local File.
    The platform does not support importing JSON, CSV, or YAML files directly. Convert them to XLSX or XLS format before importing.
  3. For Parsing Method, select Default Settings or Custom Settings. (Custom Settings lets you configure parsing rules for different formats to improve parsing results).
    Configure the parsing strategy based on your needs. If you are unsure, use the default settings. For details on Intelligent Document Parsing, LLM Parsing, and Digital Parsing, see document understanding.
    The available parsing methods depend on the selected file type (document, image, audio, and video).
    • Digital Parsing: Cannot parse illustrations and charts in files.
    • Intelligent document parsing: The parser identifies and extracts text from illustrations and then generates text summaries. These summaries, along with other non-image content, are chunked, converted into vectors, and used for knowledge base retrieval.
    • Large model document parsing: Enables agents that use a model from Select Model to answer questions about illustrations and charts in a file. Select LLM Parsing to recognize and understand this type of content.
    • Qwen-VL parsing: Supports only image formats. You can select a Qwen-VL model and use a prompt to specify the layout, elements, and content for recognition. Its other features are the same as those of large model document parsing.
    • Audio/video parsing: Performs speech recognition, video frame extraction (for videos only), and plot analysis (for videos only) on the file, then structurally aligns all audio and visual information on a timeline.
      • Speech recognition: The subtitle content parser uses Speech File Transcription to convert human speech into text. It cannot recognize music or environmental sounds, such as horns, bells, or thunder.
      • Video frame extraction: Extracts representative visual frames from the original video and generates corresponding text descriptions.
      • Plot analysis (must be enabled manually): Analyzes video content to identify specific events, marks them with timestamps, and generates corresponding text descriptions.
  4. (Optional) Configure Tags for the file.
    When you call an application by using an API, you can specify tags in the tags request parameter. When searching the knowledge base, the application first filters relevant files based on the tags to improve retrieval efficiency. For an agent application, you can set tags in the console when you debug the knowledge base.
  5. Click Confirm. The system starts parsing and importing the file. You can track the task progress on the page.
    The file will be converted into a format that Alibaba Cloud Model Studio can process. Processing may take several hours during peak times. Please be patient.
  6. After the import is complete, click Details next to the file to view it.
    Once imported into Alibaba Cloud Model Studio, files are stored as independent copies, unlinked from the original data, in the free storage space provided by the platform. This space currently has no capacity limit.
    You can only view files imported within the last 90 days. Files older than this period cannot be viewed, but they will not be deleted.
    Imported files can be used only by users in the current workspace. Alibaba Cloud Bailian will not use the files for any commercial purposes or disclose them publicly.

Dataset details

On the dataset list page, click the Details button for a dataset to view its details page. The details page contains the following tabs:
  • Overview: Shows basic information about the dataset, such as its name, description, type, creation time, and storage quota, and lists its auto-generated tools.
  • Files/Tables (platform-hosted type only): Manage the files or table data within the dataset.
  • Tools: View the details of the dataset's auto-generated tools, including parameter descriptions and online testing.
On the dataset details page, you can expand a tool, fill in the parameters, and click the Run button to test the tool's response online.

Data synchronization rules

On the Files tab of a file dataset, click Sync Data Rules > Create Sync Rule in the upper-right corner to open the rule creation page.
  1. Select a Category for the data synchronization.
  2. Select a Sync Source:
    • OSS
    • Lark
    • DingTalk
    • SharePoint
    • Yuque Sync
    Model Studio uses a service-linked role (SLR) to automatically synchronize data from OSS by accessing your buckets.
    • OSS synchronization uses a service-linked role (SLR) to access user buckets, eliminating the need to configure an AccessKey. Before first use, you must initialize the service-linked role and complete the authorization as prompted.
    • The object path must include the bucket name, for example, my-bucket/docs/ or my-bucket/docs/foo.md.
    • The current account must have read permissions for the target bucket. To grant access to Model Studio, you must also add the bailian-datahub-access tag with the value read to the target bucket. For more information, see Manage bucket tags.
    Fill in the following OSS object information:

    Parameter

    Required

    Description

    OSS region

    Yes

    Select the region where the OSS bucket is located. For a list of supported regions, see OSS Regions Overview.

    OSS object path

    Yes

    Enter the path of the OSS object to synchronize. The path must include the bucket name. For example, my-bucket/docs/ synchronizes all files in the directory, while my-bucket/docs/foo.md synchronizes a single file.

    Synchronization period

    Yes

    Select the frequency for automatic data synchronization, for example, every minute.

    Connection test

    Yes

    Click Connection Test to verify that the configured OSS path is accessible. You can submit the rule only after the test passes.

    Data tags

    No

    Configure tags for the synchronized data to help categorize it. Each tag can be up to 32 characters long and can contain Chinese characters, uppercase and lowercase letters, numbers, underscores (_), and hyphens (-). Press Enter after typing to add a tag.

  3. Configure the data tags, and then click Confirm to complete the creation.
After creation, the rule is displayed on the synchronization rule list page, where you can Enable or Disable it in the Actions column.

File dataset tools

After a file dataset is created, the system automatically generates the following two tools:

Tool name

Description

Parameters

searchFile

Searches for files by title keywords and returns their download links.

  • keyWord (string, required): Keywords from the file title.

  • maxCount (int, optional): The number of files to return. Default: 5. Maximum: 10.

getFile

Retrieves a file by using its ID and returns its download link.

  • fileId (string, required): The file ID.

Manage datasets

  • Edit: Click a dataset's card to open its details page. Click the Edit button in the upper-right corner to modify the dataset's name and description. You cannot change the dataset type or storage method after creation.
  • Copy: On the dataset list page or its details page, click the Copy button to create a new dataset with the same configuration.
  • Delete: On the dataset list page, click the More icon (···) on the target dataset card and select Delete, or click the Delete button on the dataset details page.
    Deletion requires confirmation. You cannot undo this action.
Once a knowledge base is created, you cannot change its associated dataset. The Data source field on the knowledge base edit page is read-only. To use a different dataset, you must re-create the knowledge base.

Use a data connector in an agent

After you create a connector, add its tools to your agent. The agent can then automatically call these tools in conversations to query external data.
  1. On the agent configuration page, click Skills in the left pane and find the Data Connector section.
  2. Click the + button in the Data Connector section. In the Select Data Connector dialog box:
    1. Browse or search for the target connector. You can filter by connector type.
    2. Click Add next to the target connector.
  • Once added, the connector's tools automatically appear in the configuration list. You can click the settings button next to a tool to adjust its parameters.
  • Publish the agent so it can automatically call these tools in conversations.

Use a data connector in a workflow

After you create a connector, you can add a data connector node to your workflow. This allows the workflow to call connector tools to query external data and pass the results to downstream nodes.
  1. On the workflow canvas configuration page, expand the node library on the left. Under the tools category, find the data connector node and drag it onto the canvas.
  2. In the Select data connector dialog box:
    • Browse or search for the target connector. You can filter by connector type, such as File, Table, MySQL, PostgreSQL, Yuque, or OSS.
    • Expand the connector, select the tool you want to use (such as searchFile or getFile), and then click OK.
  3. Configure node input: In the Input section of the node configuration panel, configure the tool's input parameters. Map outputs from upstream nodes or built-in variables to the required parameters, such as fileId.
  4. Connect nodes: Connect the data connector node to upstream and downstream nodes to ensure the correct data flow.
  5. The node's output is a result object with two fields: content (Array<Object>, the returned content) and isError (Boolean, indicates whether an error occurred). Downstream nodes can reference this output.
Each data connector node in a workflow can use only one tool. If you need to use multiple tools from the same connector, add multiple data connector nodes.

References

To create a knowledge base and import content from a data source for retrieval, see Create and use a knowledge base. For guidance on configuring and using applications, see Introduction to application types.

FAQ

Permissions and security

The root account must grant the Administrator permission to the RAM user. For more information, see Page permissions.
Data connectors transmit data through public network links, but the entire transmission uses HTTPS/TLS encrypted channels. Data is in ciphertext during transmission and cannot be interpreted even if the link is intercepted. After data reaches the platform, it is written directly to Alibaba Cloud internal network secure storage, which cannot be accessed from the public network. The public network only serves as a transport medium—data is not persisted on the public network.After import, files are stored as independent copies (unlinked from original data) in the platform's secure storage. Imported files are accessible only to users in the current workspace. Alibaba Cloud Model Studio does not use the data for any commercial purposes or make it publicly available.If you still have concerns about public network transmission, you can choose the following options for enhanced security:
  • Direct file upload: Upload local files through the file connector. Data does not pass through public network links and is written directly to Alibaba Cloud internal network secure storage.
  • Private network data source: For Alibaba Cloud RDS for MySQL or self-managed MySQL databases, you can select private network (VPC) connection. Data is transmitted through the internal network, offering better security and performance. See the Private network data source section above.
Model Studio Standard and Enterprise editions have identical data security mechanisms: both provide user-level isolation, workspace member-only access, and HTTPS/TLS encrypted transmission. The difference between the two editions is only in storage capacity and feature limits, not security.

Import files from OSS

The first time you import files from OSS, you must authorize Alibaba Cloud Bailian to access your OSS resources. The authorization process differs for a root account and a RAM user.
  • Authorize a root account
  • Authorize a RAM user
  1. On the Import data page, click Authorize Now.
  2. In the dialog box that appears, click Confirm Authorization. The system automatically creates an OSS service-linked role to allow Alibaba Cloud Bailian to access your OSS resources.
    This change usually takes effect within seconds but can be delayed during peak hours.
  3. Add the tag bailian-datahub-access with the value read to the target OSS bucket.
    This tag identifies the buckets that Alibaba Cloud Bailian can access. Alibaba Cloud Bailian cannot access buckets without this tag.
    1. Go to the OSS console. In the left-side navigation pane, click Bucket list and find the target bucket.
    2. Hover over the Tag icon next to the bucket name and click Modify (if no tags have been set) or Go to edit.
    3. On the bucket tags page, click Create tag (if no tags have been set) or Settings.
    4. Click Tag, add a tag with the key bailian-datahub-access and the value read, and then click Save.
  4. Return to the Import Data page, reselect the target bucket, and try the import again.
    Note: Alibaba Cloud Bailian does not support accessing files in the root directory of a bucket. You must select an existing subdirectory or create a new one for Alibaba Cloud Bailian to access.
This error usually occurs because the root account has not activated OSS. To resolve this issue:
  1. Sign in to the OSS console as the root account and activate OSS when prompted.
  2. Return to the Alibaba Cloud Bailian page and authorize again.

Import MySQL data

  • China (Hangzhou)
  • China (Shanghai)
  • China (Shenzhen)
  • China (Heyuan)
  • China (Guangzhou)
  • China (Qingdao)
  • China (Beijing)
  • China (Zhangjiakou)
  • China (Hohhot)
  • China (Ulanqab)
  • China (Chengdu)
  1. As the root account, attach the following three system policies to the RAM user: AliyunBailianFullAccess, AliyunEventBridgeFullAccess, and AliyunRDSReadOnlyAccess. For detailed instructions, see Manage permissions for a RAM user.
  2. Using the root account, grant the RAM user the permission to create service-linked roles.
    1. Sign in to the RAM console as the root account. In the left-side navigation pane, choose Permissions > Policies, and then click Create policy.
    2. In the Script Editor, enter the corresponding content from the following script into the Effect, Action, Resource, and Condition fields, and then click OK.
{
    "Version": "1",
    "Statement": [
        {
            "Action": "ram:CreateServiceLinkedRole",
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}
  1. Enter the policy name CreateServiceLinkedRole and click OK.
  2. In the left-side navigation pane, choose Identities > Users. Find the target RAM user in the list and click Add permissions in the Actions column.
  3. From the Policies list, select the CreateServiceLinkedRole policy that you just created, and then click OK. This grants the RAM user permission to create service-linked roles.
  4. After completing these steps, return to the Create Data Source page as the RAM user and create the EventBridge service-linked role again.
  • ApsaraDB RDS for MySQL
  • Self-managed MySQL
The image is for illustration only. Recommended values vary based on your table's data volume. If an item is not mentioned, no change is required.Perform these steps as your root account.
  • Configure the local log retention period
    1. Go to the RDS console. In the left-side navigation pane, click Instances, and then click the ID of the relevant RDS instance. In the left-side navigation pane, click Backup and recovery. On the Backup strategy tab, find the Retention period setting.
    2. Set the Retention period to the value recommended in the prompt.
  • Configure wait_timeout
    1. Go to the RDS console. In the left-side navigation pane, click Instances, and then click the ID of the relevant RDS instance. In the left-side navigation pane, click Parameter settings. On the Editable parameters tab, find the wait_timeout parameter.
    2. Set the parameter to the value recommended in the prompt.
Agent Development
Agent Context
Agent Governance & Evolution
Store
Application Marketplace
Use Cases
Support