Skip to main content

Automatic Document Tagging Using Large Language Models

Automatically extract and write back document tags using the Knowledge Q&A service, enabling an end-to-end workflow—from document ingestion to tag-based retrieval.

This practice leverages the retrieval and generation capabilities of the Knowledge Q&A service. A large language model (LLM) automatically extracts tags from all document chunks and writes them back to the document—establishing an end-to-end pipeline from ingestion → automatic tagging → tag-based retrieval. The entire process is completed in the console and is fully reproducible.

Solution Overview

Background

Manual document tagging suffers from three key issues:
  • Inability to keep pace with ingestion speed: During bulk document ingestion, manual tagging lags far behind ingestion throughput.
  • Latency between ingestion and usability: Documents remain unavailable for tag-based search and distribution until manually tagged.
  • Inconsistent tag standards: Human annotators interpret content differently, leading to inconsistent tags across similar documents—and degrading retrieval quality.
Automated tag extraction by LLMs—guided by document content—enables tagging to occur concurrently with ingestion, while consistent prompting ensures uniform tag semantics.

Applicable Scenarios

ScenarioDescriptionValue of Automatic Tagging
Enterprise Document ManagementTag contracts, project documentation, and technical manuals with attributes such as department, project, topic, and classification levelImproves search efficiency and supports permission control and compliance traceability
Knowledge Asset ConsolidationTag technical documents, research reports, and industry whitepapers with domain, tech stack, and use casesStructures scattered materials, enabling precise tag-based Q&A and recommendation
Marketing Asset ManagementTag product manuals, marketing plans, and customer case studies with product line, industry, and scenarioEnables rapid asset discovery and supports tag-based filtering and reuse

Technical Principle

The Knowledge Q&A service is fundamentally a RAG (Retrieval-Augmented Generation) system: it retrieves relevant chunks first, then uses an LLM to generate answers based on those chunks. By shifting the generation objective from answering questions to extracting tags, we achieve automatic tagging: assign a unique temporary tag to the target document, filter retrieval exclusively to that document’s chunks via tag filtering, prompt the LLM to output structured tags, and finally write them back. 
Original Document ──→ Import into Knowledge Base (parsing + chunking + vectorization)
                  ──→ Assign a unique temporary tag (auto_tag_temp)
                  ──→ Knowledge Q&A Service (tag filtering selects the unique tag)
                  ──→ Disable the ranking model; set recall count to maximum
                  ──→ LLM + tagging prompt (outputs structured tags)
                  ──→ Write back to the document's tags field
                  ──→ Tag-based retrieval
Three critical implementation points:
  • Single-document retrieval: Bailian’s tag filtering operates on the tags field—not document IDs. Assigning a unique tag (e.g., auto_tag_temp) to the target document ensures only its chunks are retrieved.
  • Maximize coverage: Disable the ranking model and set recall count to maximum so as many chunks as possible enter the LLM context.
  • Structured output: Use prompt engineering to constrain LLM output to parseable, directly writable tag arrays.
The maximum number of chunks retrievable per Knowledge Q&A request is capped at 20. If a document has ≤20 chunks, full coverage is achieved in one call. For documents exceeding 20 chunks, either split the document or perform batched tagging. Sample documents used in this demo typically contain fewer than 20 chunks.

Step-by-Step Implementation

This practice demonstrates using the Bailian Technical Documentation Sample (92 technical documents about the Bailian platform), taking "First Call to Tongyi Qwen API" as the example document. You may substitute your own documents. The following tags will be written back automatically—no pre-creation required:
TagDescriptionExample Values
productProduct the document belongs toproduct_Bailian
topicCore subject of the documenttopic_API_call
difficultyEntry-level / Intermediate / Expertdifficulty_Beginner
keywordKey terms (multi-value)keyword_Environment_Variable, keyword_curl
Bailian tags are simple string arrays—no schema or data types. Supported characters: Chinese, English letters (case-sensitive), digits, underscores (_), and hyphens (-). Colons (:), spaces, and other special characters are not allowed. Each tag must be ≤32 characters; each document supports up to 100 tags. Therefore, semantic tags follow the field_name_value format (e.g., product_Bailian). Multi-value tags (e.g., keywords) are represented as separate strings.

1. Create a Knowledge Base and Import Documents

Navigate to Data Ingestion → Knowledge Management, click Create, and select Premium Edition (required for higher recall limits). Follow the three-step wizard:
  1. Basic Info: Enter knowledge base name and description. Select Document Search as knowledge base type and Basic Document Q&A as use case.
  2. Select Data: Configure step-by-step: choose connector (default: File Connector), select Upload Files as data source, configure categories, and upload the sample ZIP file in the File Upload section.
  3. Index Settings: Keep defaults and complete creation.
After creation, go to the knowledge base’s Document List, and wait until document status changes to Parsing Complete (system auto-performs parsing, chunking, and vectorization).
The Add Tag option at the bottom of the Select Data step applies tags batch-wise to all uploaded files—and cannot isolate individual documents. Unique per-document tags (e.g., auto_tag_temp) are added separately in the next step. Tags are free-text; no prior business tag definition is needed.
See Create a Knowledge Base and Document Management for details.

2. Assign a Unique Temporary Tag to the Target Document

Since tag filtering is our mechanism for single-document retrieval, first assign a unique temporary tag (e.g., auto_tag_temp) to the document you wish to tag:
  • Console: Go to Data Ingestion → Knowledge Management, click View Details for the knowledge base created above, locate the target document in the Document List, click More → Tags, enter auto_tag_temp in the tag management modal, press Enter, then confirm.
  • API: Call batchUpdateFileTag to write ["auto_tag_temp"] into the document’s tags array.

3. Configure the Knowledge Q&A Service

  1. Navigate to Knowledge Services → Knowledge Q&A, click Create, and bind the knowledge base from Step 1.
  2. Select a generation model. Tag extraction demands strong comprehension and instruction-following capability—use a high-performance model (e.g., qwen3.7-plus).
  3. Click Expand Knowledge Base Configuration, and set retrieval parameters as follows:
ParameterSettingReason
Tag FilteringInput or select the unique tag auto_tag_tempEnsures only chunks from this document are retrieved
Initial Vector Retrieval TopKIncrease (≥ number of document chunks; default 50)Maximizes inclusion of this document’s chunks in candidate pool
Initial Keyword Retrieval TopKIncrease (≥ number of document chunks; default 50)Covers keyword-based retrieval path
Ranking ModelSelect Do Not Use ModelPrevents reranking from discarding relevant chunks
Max Recall CountSet to maximum 20Maximizes number of chunks passed to final LLM
Core goal: maximize LLM’s exposure to the full document. Always set ranking model to Do Not Use Model, and max recall count to 20. Due to this hard limit, documents with >20 chunks require splitting or batched tagging.

4. Configure the Tagging Prompt

Paste the tagging prompt template into the Prompt input box on the configuration page. The service automatically concatenates retrieved chunks into context—no placeholder syntax needed. The prompt must explicitly instruct the model to output a parseable, writeable field_value tag array: 
# Task
You are a document content understanding assistant. The provided reference material consists of multiple chunks from the *same document*, selected to cover its content as comprehensively as possible. Treat these chunks as sections of a single, unified document—and extract tags accordingly.

# Behavioral Requirements
1. Synthesize information across *all* chunks—do not rely on isolated segments alone.
2. All tag values must derive *exclusively* from the reference material—no external knowledge or subjective inference.
3. Omit any dimension whose value is not explicitly stated or clearly implied in the content.
4. Do not repeat text verbatim or explain reasoning—output *only* the tag array.

# Tag Dimensions
- product: Product the document belongs to (e.g., Bailian, Tongyi Qwen)
- topic: Primary subject (e.g., API_call, model_inference, data_import)
- difficulty: Beginner / Intermediate / Expert
- keyword: 1–5 key terms from the document

# Output Format
- Express each tag as `field_name_value`; multi-value dimensions (e.g., keyword) yield multiple tags.
- Only allow Chinese characters, letters, digits, underscores (`_`), and hyphens (`-`). No colons, spaces, or other symbols.
- Output *exactly one JSON string array*, with no Markdown, comments, or extraneous text. Example:
["product_Bailian", "topic_API_call", "difficulty_Beginner", "keyword_Environment_Variable", "keyword_curl"]

5. Execute Tagging via Console

In the Debug panel on the right side of the configuration page, enter a tagging instruction (e.g., Please extract tags for this document) and send. Verify that the LLM’s returned tag array matches expectations. For batch tagging multiple documents: after confirming correctness, click Publish to obtain the service ID (agent_id). Then invoke the Knowledge Q&A API to submit tagging requests (responses are SSE-streamed; see interface docs for parsing). Finally, proceed to Step 6 to write tags back. For batch workflows, see Notes for ordering guidance.

6. Write Tags Back & Enable Tag-Based Retrieval

Once the LLM outputs the tag array, call batchUpdateFileTag to persist tags (using OVERWRITE mode replaces the temporary auto_tag_temp): 
curl -X POST 'https://{workspace_id}.cn-beijing.maas.aliyuncs.com/api/v1/connector/dash/batchUpdateFileTag' \
  -H 'Authorization: Bearer $API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "fileInfos": [
      {
        "fileId": "file-xxx",
        "tags": ["product_Bailian", "topic_API_call", "difficulty_Beginner", "keyword_Environment_Variable", "keyword_curl"]
      }
    ],
    "updateMode": "OVERWRITE"
  }'
Retrieve fileId via listFile, or manually write tags via More → Tags in the console’s document list. After writing, enable precise retrieval by selecting corresponding tags under Knowledge Base Configuration → Tag Filtering, in either Knowledge Search or Q&A services.

Notes

Tags Are String Arrays

Bailian tags are untyped string arrays. Allowed characters: Chinese, English letters (case-sensitive), digits, underscores (_), hyphens (-). Colons (:), spaces, and other symbols are forbidden. Max length per tag: 32 chars; max tags per document: 100. Semantic meaning is encoded via field_name_value (e.g., product_Bailian). Multi-value fields (e.g., keywords) become separate strings. Tags are free-text—no pre-registration required.

Single-Document Retrieval Relies on Unique Tags

Bailian’s retrieval and Q&A do not support document-ID filtering—only tags-based filtering. Thus, single-document tagging requires assigning a unique temporary tag (e.g., auto_tag_temp) per document. Tags added during the Select Data step apply in bulk to all uploaded files—and cannot isolate individual documents.

Constrained by Max Recall Limit

The Knowledge Q&A service caps retrievable chunks per request at 20. Documents with >20 chunks cannot be fully processed in one call—split them or use batched tagging.

Tag Filtering Is a Static, Service-Level Configuration

Tag filtering in Q&A services is fixed upon publishing the configuration—it does not vary per request. For batch tagging multiple documents, choose one of two approaches:
ApproachHow ToUse Case
Shared Temporary TagAssign same temp tag (e.g., auto_tag_temp) to all docs; tag sequentially; overwrite with business tags afterwardSmall batches, serial execution
Dedicated ServicesCreate one Q&A service per document (or group), each configured with its own tag filterLarge-scale parallel tagging

Frequently Asked Questions

Why does tagging only cover part of the document? Confirm ranking model is disabled and both TopK and max recall are maximized. For documents with >20 chunks, split or batch. Why does the model output extra text—not just the tag array? Strengthen prompt constraints requiring only the JSON array. Before parsing, extract the first [...] substring from the response. Why does tag write-back fail? Check for unsupported characters (colons, spaces) and ensure no tag exceeds 32 characters.
Agent Development
Agent Context
Agent Governance & Evolution
Store
Application Marketplace
Use Cases
Support