DashScopeJsonNodeParser is a text chunking model from Alibaba's Tongyi Lab that allows developers to use the LlamaIndex framework to chunk documents parsed by DashScopeParse.
Get started
Prerequisites
First, log on to https://bailian.console.aliyun.com/cn-beijing/ to obtain your API key. To specify a workspace, you must also obtain the workspace ID.
- An API key has been created. For more information, see Obtain and configure an API key.
- A workspace has been created. For more information, see Workspace management.
Sample code
- DashScopeJsonNodeParser is primarily used with DashScopeParse.
Parameters
Parameter value | Default value | Description |
try_count_limit | 10 | The maximum number of retry attempts. |
chunk_size | 500 | The size of each chunk to process. |
overlap_size | 100 | The overlap size between consecutive chunks. |
separator | " |,|,|。|?|!|\n|\?|\!" | The separators used to split text. |
language | cn | The language for the lexical analyzer. Accepts "cn" for Chinese, "en" for English, and "any". Note: The "any" pattern is slower. |