Skip to main content

Build an Alibaba Cloud Model Studio cloud knowledge base and use the cloud knowledge index service with DashScopeCloudIndex (DashScopeCloudRetriever)

Use DashScopeCloudIndex and DashScopeCloudRetriever to easily manage cloud document indexes in Alibaba Cloud Model Studio and build retrievers with the LlamaIndex framework.

Prerequisites

Build a cloud knowledge base from local files

image
Before you run this example, note the following:
  1. Install the following dependencies:
pip install llama-index-readers-dashscope
pip install llama-index-indices-managed-dashscope
  1. Prepare the files for parsing. This example uses the Alibaba Cloud Model Studio Product Introduction (Fictional).zip file. You can upload and parse local files using the DashScopeParse module.
    Intelligent document parsing in DashScopeParse supports the PDF, DOC, DOCX, TXT, MD, PPT, PPTX, XLS, and XLSX formats. A single document can be up to 100 MB or 1,000 pages. You can upload a maximum of 200 files at a time.Only PDF, DOC, and DOCX files are intelligently parsed. Other file types are uploaded to the cloud as-is without parsing.
Go to Application Data > Files to obtain the category ID (category_id).
image
If you do not set a category ID (category_id) when using DashScopeParse, the default category in Application Data-Files is used.
Example code: 
import os
from llama_index.readers.dashscope import DashScopeParse
from llama_index.indices.managed.dashscope import DashScopeCloudIndex

# You can use the following command to set a temporary environment variable for DASHSCOPE_WORKSPACE_ID (workspace ID).
# os.environ["DASHSCOPE_WORKSPACE_ID"] = "llm-xxx<Replace with your workspace_id>"
if "DASHSCOPE_WORKSPACE_ID" not in os.environ or os.environ["DASHSCOPE_WORKSPACE_ID"] is None:
    raise ValueError("DASHSCOPE_WORKSPACE_ID is not set. Cannot operate DashScope Cloud Index.")

# Set the list of files to upload.
file_list = [
    "./docs/Alibaba Cloud Model Studio Tablet Series Introduction.pdf",
    "./docs/Alibaba Cloud Model Studio Phone Series Introduction.docx",
    "./docs/Alibaba Cloud Model Studio Smart Speaker Introduction.txt"
]
# Upload files and perform intelligent parsing in the cloud.
documents = DashScopeParse(
    category="cate_xxx<Replace with the category ID. If not set, the default category is used.>",
    # More parameters are supported.
).load_data(file_path=file_list)
print("Document upload and intelligent parsing completed.")

# Perform intelligent chunking and indexing in the cloud.
index = DashScopeCloudIndex.from_documents(
    documents,
    name="my_first_index",
    # More parameters are supported.
)
# The cloud support library supports the following operations.
# Add more documents.
# index._insert(documents)
# Delete some documents.
# index.delete_ref_doc([documents.doc_id])
print("Cloud knowledge base built.")

# Initialize the retrieval engine. The retriever is a DashScopeCloudRetriever class.
retriever = index.as_retriever(
    # More parameters are supported.
    # dense_similarity_top_k = 100,
    # sparse_similarity_top_k = 100,
    # enable_reranking = True,
    # ...
)
print("===========================================================================================================")
# Perform a vector database retrieval.
nodes = retriever.retrieve("What do you know about the Alibaba Cloud Model Studio phone?")
# Display the first retrieved TextNode vector.
print(nodes[0])
Sample output: 
Document upload and intelligent parsing completed.
Index my_first_index created successfully!
Cloud knowledge base built.
===========================================================================================================
Hit test: What do you know about the Alibaba Cloud Model Studio phone?
First result from the hit test:
Node ID: llm-xxx_xx_file_xx
Text: Welcome to the forefront of future technology. Explore our meticulously crafted smartphone series, each designed to fulfill your infinite imagination for a tech-driven life. Alibaba Cloud Model Studio X1 ——Enjoy the ultimate visual experience: Equipped with a 6.7-inch 1440 × 3200 pixel ultra-clear screen and a 120Hz refresh rate, it delivers a smooth visual experience. The combination of 256 GB mass storage and 12 GB RAM handles large games and multitasking with ease. A 5000mAh long-lasting battery, plus an ultra-sensitive quad-camera system, captures every wonderful moment of life. Reference price: 4599-4999 Tongyi Vivid 7 ——A new experience in smart photography: Features a 6.5-inch 1080 × 2400 pixel full screen. The AI smart photography feature ensures every photo displays professional-grade color and detail. 8 GB RAM and 128 GB storage ensure smooth operation, and the 4500mAh battery meets daily needs. Side fingerprint unlock is convenient and secure. Reference price: 29...
Score:  0.567

Use an Alibaba Cloud Model Studio cloud knowledge base directly

image
Example code: 
import os
from llama_index.indices.managed.dashscope import DashScopeCloudIndex

# You can use the following command to set a temporary environment variable for DASHSCOPE_WORKSPACE_ID (workspace ID).
# os.environ["DASHSCOPE_WORKSPACE_ID"] = "llm-xxx<Replace with your workspace_id>"
if "DASHSCOPE_WORKSPACE_ID" not in os.environ or os.environ["DASHSCOPE_WORKSPACE_ID"] is None:
    raise ValueError("DASHSCOPE_WORKSPACE_ID is not set. Cannot operate DashScope Cloud Index.")

# A knowledge base named "my_first_index" must exist in the workspace.
index = DashScopeCloudIndex("my_first_index")
print("Cloud knowledge base built.")

# Initialize the retrieval engine. The retriever is a DashScopeCloudRetriever class.
retriever = index.as_retriever(
    # More parameters are supported.
    # dense_similarity_top_k = 100,
    # sparse_similarity_top_k = 100,
    # enable_reranking = True,
    # ...
)
print("===========================================================================================================")
# Perform a vector database retrieval.
nodes = retriever.retrieve("What do you know about the Alibaba Cloud Model Studio phone?")
# Display the first retrieved TextNode vector.
print(nodes[0])
Sample output: 
Cloud knowledge base built.
===========================================================================================================
Hit test: What do you know about the Alibaba Cloud Model Studio phone?
First result from the hit test:
Node ID: llm-xxx_xx_file_xx
Text: Welcome to the forefront of future technology. Explore our meticulously crafted smartphone series, each designed to fulfill your infinite imagination for a tech-driven life. Alibaba Cloud Model Studio X1 ——Enjoy the ultimate visual experience: Equipped with a 6.7-inch 1440 × 3200 pixel ultra-clear screen and a 120Hz refresh rate, it delivers a smooth visual experience. The combination of 256 GB mass storage and 12 GB RAM handles large games and multitasking with ease. A 5000mAh long-lasting battery, plus an ultra-sensitive quad-camera system, captures every wonderful moment of life. Reference price: 4599-4999 Tongyi Vivid 7 ——A new experience in smart photography: Features a 6.5-inch 1080 × 2400 pixel full screen. The AI smart photography feature ensures every photo displays professional-grade color and detail. 8 GB RAM and 128 GB storage ensure smooth operation, and the 4500mAh battery meets daily needs. Side fingerprint unlock is convenient and secure. Reference price: 29...
Score:  0.567

API reference

DashScopeParse API Reference DashScopeCloudIndex: API reference
Managed Agent API
Flow Agent API
Sandbox API
RAG API
Long-term Memory
ParseX API
Security API
Framework Integration
Application components
More
Assistant API (Deprecated)