Skip to main content

DashScopeJsonNodeParser

DashScopeJsonNodeParser is a text chunking model from Alibaba's Tongyi Lab that allows developers to use the LlamaIndex framework to chunk documents parsed by DashScopeParse.

Get started

Prerequisites

First, log on to https://bailian.console.aliyun.com/cn-beijing/ to obtain your API key. To specify a workspace, you must also obtain the workspace ID. Then, install the DashScopeJsonNodeParser package (Python >=3.9 and <=3.12).
pip install llama-index-core
pip install llama-index-node-parser-dashscope

Sample code

import json
import os
from llama_index.node_parser.dashscope import (
    DashScopeJsonNodeParser,
)
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.schema import Document

# Set your Dashscope API key in the environment
os.environ["DASHSCOPE_API_KEY"] = "your_api_key_here"

documents = [
    # Prepare your documents obtained from the Dashscope reader
]

# Initialize the DashScope JsonNodeParser
node_parser = DashScopeJsonNodeParser(
    chunk_size=100, overlap_size=0, separator=" |,|,.|?|!|\n|?|!"
)

# Set up the ingestion pipeline with the node parser
pipeline = IngestionPipeline(transformations=[node_parser])

# Process the documents and print the resulting nodes
nodes = pipeline.run(documents=documents, show_progress=True)
for node in nodes:
    print(node)
Note:
  • DashScopeJsonNodeParser is primarily used with DashScopeParse.

Parameters

Parameter value

Default value

Description

try_count_limit

10

The maximum number of retry attempts.

chunk_size

500

The size of each chunk to process.

overlap_size

100

The overlap size between consecutive chunks.

separator

" |,|,|。|?|!|\n|\?|\!"

The separators used to split text.

language

cn

The language for the lexical analyzer. Accepts "cn" for Chinese, "en" for English, and "any". Note: The "any" pattern is slower.

Managed Agent API
Flow Agent API
Sandbox API
RAG API
Long-term Memory
ParseX API
Security API