Online Policy Distillation Training Overview

> ## Documentation Index > Fetch the complete documentation index at: https://docs.bailian.console.aliyun.com/llms.txt > Use this file to discover all available pages before exploring further. # Online Policy Distillation Training Overview > Bailian Online Policy Distillation (OPD) training overview — basic concepts, training method selection, end-to-end flow, environment preparation, and job submission This article helps you understand the basic concepts, Applicable Scenario, and end-to-end process of Bailian Online Policy Distillation (OPD) Training. This article provides an overview and key explanations; for detailed operation steps of each step, see the corresponding Docs:

Docs	Content
[Model OPD Development](opd-model-development-guide.md)	Pure distillation and Reward function Development, Data format, function testing
[Agentic OPD Development](opd-agentic-development-guide.md)	Custom Rollout (passing tools parameter to produce structured tool\_call), Agentic Reward Development
[Training Configuration](opd-training-config.md)	Submission Method, teacher model configuration, hyperparameters, Environment Variable, tuning and pitfalls
[Observability configuration](opd-observable-config.md)	Tracing integration, Custom metrics, metric dictionary, effect evaluation, troubleshooting decision tree

OPD (Online Policy Distillation) is a Training method that uses a stronger teacher model to supervise a student model token by token. The teacher model provides a probability distribution for each token in the student model's Output, and the student model generates the Output itself, with a guidance signal at each step. The goal is to transfer the capabilities of a large model on specific tasks to a smaller model of the same family, reducing inference cost while maintaining effectiveness. Implementation OPD Training each step: 1. **Student generation**: student model generates a segment of Output 2. **Teacher gives distribution**: teacher model provides a probability distribution for each token position in the Output (i.e., the teacher's belief of the probability of each token appearing at that position) 3. **Compute difference**: student model compares its own probability distribution with the teacher's distribution, computing the difference at each token position (KL divergence, a metric measuring the difference between two probability distributions) 4. **Update toward**: use this as loss to update parameters, making each token move toward the teacher model The teacher model only provides probability distributions; it does not generate text or participate in dialogue or tool calls. Each token position has a guidance signal, rather than waiting until the entire Output is finished to give a single overall score. Differences between OPD and Fine-tuning, traditional model distillation

Dimension	Fine-tuning (SFT)	Traditional model distillation	Online Policy Distillation (OPD)
Training Data	Manual Annotation	Teacher model pre-generates Output	Student model generates
Guidance method	Fixed labels	Teacher model complete Output	Teacher model token-by-token probability distribution
Student model participates in generation	No	No	Yes
Error correction	None	None	Token-by-token correction

## Full Process Overview OPD Training from environment preparation to model deployment, Total 5 steps: 1. Environment preparation — Install SDK, Obtain API Key, complete authorization 2. Prepare data and code — JSONL data, pure distillation does not need to write functions 3. Submit Training Task — Pass `teacher_model` to trigger distillation, pure distillation with zero code 4. Training Status Observation — Check distillation loss, teacher model signal health, validation set metrics 5. Publish and Deploy Model — Final Checkpoint auto-published, called via API after deployment ### Working Mechanism The data flow for each training round is as follows: ```plaintext 训练数据（用户问题 + 参考答案） │ ▼ 学生模型生成：学生模型按当前策略生成候选输出 │ ▼ 教师模型前向：教师模型对学生模型输出逐 token 给概率分布 │ ▼ 蒸馏更新：学生模型最小化自身分布与教师模型分布差异（KL 散度） │ ▼ （可选）Reward 评分：启用 Reward 时叠加任务奖励信号 │ ▼ 循环迭代：用更新后的模型重新采样，逐轮向教师模型靠拢 ``` The teacher model only provides probability distributions, does not generate text, and does not participate in dialogue or tool calls. Pure distillation is solely guided token by token by the teacher model; when Reward is simultaneously enabled, the teacher model distribution and task reward jointly drive the student model. ## Model and Training Unit The teacher model must be a model already enabled in Model Studio, in the same series as and stronger than the student model.

Teacher Model	Student Model	Invite	Recommended Training Unit
qwen3.5-122b-a10b	qwen3.5-27b	^Preview	IV Type (MTU4) × 24
qwen3.5-397b-a17b	qwen3.5-27b	^Preview	IV Type (MTU4) × 24
qwen3.5-122b-a10b	qwen3.5-2b	^Preview	IV Type (MTU4) × 24
qwen3.5-27b	qwen3.5-2b	^Preview	IV Type (MTU4) × 24
qwen3.5-35b-a3b	qwen3.5-2b	^Preview	IV Type (MTU4) × 24
qwen3.5-397b-a17b	qwen3.5-2b	^Preview	IV Type (MTU4) × 24
qwen3.5-122b-a10b	qwen3.5-35b-a3b	^Preview	IV Type (MTU4) × 24
qwen3.5-397b-a17b	qwen3.5-35b-a3b	^Preview	IV Type (MTU4) × 24
qwen3.5-122b-a10b	qwen3.5-4b	^Preview	IV Type (MTU4) × 24
qwen3.5-27b	qwen3.5-4b	^Preview	IV Type (MTU4) × 24
qwen3.5-35b-a3b	qwen3.5-4b	^Preview	IV Type (MTU4) × 24
qwen3.5-397b-a17b	qwen3.5-4b	^Preview	IV Type (MTU4) × 24
qwen3.5-122b-a10b	qwen3.5-9b	^Preview	IV Type (MTU4) × 24
qwen3.5-27b	qwen3.5-9b	^Preview	IV Type (MTU4) × 24
qwen3.5-35b-a3b	qwen3.5-9b	^Preview	IV Type (MTU4) × 24
qwen3.5-397b-a17b	qwen3.5-9b	^Preview	IV Type (MTU4) × 24

Teacher Model	Student Model	Invite	Recommended Training Unit
-	-	-	-

## Quick Start Complete the 5-step full flow with a pure distillation basic example; zero code is enough to start training. Pure distillation requires no function writing and is solely guided token by token by the teacher model in training the student model. ### Environment Preparation Environment preparation requires completing the following: Obtain Sample Code Package, API Key, service authorization, SDK installation, and core Environment Variable. Workspace Directory Obtain from the[Sample Code Package](https://g-adoc.alcasset.com/media/maas_docs/sfm/zh/files/6a4b3c2d1e0f92ef.zip), containing wheel (Python packaged install File), function code, Data, dependencies, and submission script, structured as follows; the complete implementation of obtaining and submission script is in[Training Configuration](opd-training-config.md)： ```plaintext workspace/ ├── dashscope-*.whl # SDK wheel：示例附带的 dashscope 安装包，本机与远端均按此版本安装 ├── functions/ # Rollout/Reward 函数代码目录；纯蒸馏 functions=None 可留空 ├── data/ # JSONL 训练/验证数据；每行一个样本（messages + rollout_extra） ├── requirements.txt # 函数依赖清单；FC_LAYER_USED=true 时注册阶段据此构建 layer，跳过冷启动 pip └── submit_job.py # 提交脚本：调用 AgenticRL().run() 完成注册→上传→提交三步 ``` API Key From[Model Studio Console API-KEY Page](https://bailian.console.aliyun.com/?tab=app#/api-key)Obtain , set as Environment Variable `DASHSCOPE_API_KEY`。 Service Authorization On first use, in the[Model Fine-tuning Console](https://bailian.console.aliyun.com/cn-beijing/model/tuning)One-Click Authorization for three cloud services: ARMS (Application Real-time Monitoring Service, hosting OpenTelemetry metric reporting), FC, and SLS (Log Service). OpenTelemetry is an open-source observability data standard, with ARMS providing hosting and visualization, used respectively for metric reporting, function running, and log collection. SDK Install using the wheel file bundled in the example directory (version updates with the example, current 1.27.5): ```plaintext # 安装示例目录附带的 dashscope SDK wheel 文件（版本随示例更新，当前 1.27.5） # 该 wheel 同名相对路径需填入 FC_PYPI_LIB，供远端函数运行时安装 python -m pip install ./dashscope-*.whl ``` Three core Environment Variable: - `DASHSCOPE_API_KEY`(required): Authentication. - `FC_PYPI_LIB`(required): specifies the SDK package installed in the function runtime, fill in the wheel relative path (resolved remotely, local absolute path cannot hit); pure distillation (functions=None) does not involve function runtime, this item can be skipped. - `FC_LAYER_USED`(Optional, Default true): when true, builds layer based on requirements.txt during registration phase, skipping cold-start pip install; when false, installed by remote runtime cold-start. ```plaintext export DASHSCOPE_API_KEY="sk-your-api-key" # 鉴权密钥，从百炼控制台 API-KEY 页获取（必需） export FC_PYPI_LIB=./dashscope-1.27.5-py3-none-any.whl # 远端函数运行时安装的 SDK 包，填 wheel 相对路径；纯蒸馏可不设（必需，纯蒸馏除外） export FC_LAYER_USED=true # true：注册阶段按 requirements.txt 构建 layer，跳过冷启动 pip 安装；false：远端冷启动安装 ``` `FC_PYPI_LIB` Fill in the actual wheel File name (version number updates with the example). Before submission, the example script validates whether the installed SDK's `AgenticRL.run` accepts the `teacher_model` Parameter, and directly errors out if not accepted. Other Environment Variable (control plane endpoint, inference endpoint, log level, etc.) see[Training Configuration](opd-training-config.md)。 ### Prepare Data and Code JSONL one sample per line, containing `messages` and `rollout_extra` two fields.`messages` is the dialogue input (ChatML Format),`rollout_extra` stores the Reference Answer Or other business Data, consumed when Reward/Custom Rollout is enabled. ```plaintext { "messages": [ // 对话输入，ChatML 格式；每条 message 含 role 与 content {"role": "user", "content": "<您的业务问题>"} ], "rollout_extra": {"solution": "<参考答案或其他业务数据>"} // 参考答案/业务数据，供 Reward 或自定义 Rollout 启用时消费；纯蒸馏仅透传不消费 } ``` For quick validation, dozens of entries suffice; for formal training, hundreds to thousands of entries, depending on task complexity. In pure distillation, `rollout_extra` is only data pass-through and not consumed. ### Submit Training Task Entry `AgenticRL().run(...)`，pass `teacher_model` then triggers distillation.`run()` Done in one step: register function to FC → Upload Training Data → Submit training task. Pure distillation does not register a function. The following is a minimal submission code example, with comments explaining each Parameter's meaning. Complete implementation see[Training Configuration](opd-training-config.md)'s submit\_job.py. teacher\_model is a placeholder value, actually defined by you. ```plaintext expandable import asyncio from dashscope.finetune.agentic_rl import AgenticRL # OPD 提交客户端 from dashscope.finetune.reinforcement import ( TrainingDataset, ValidationDataset, DataSourceType, # 训练/验证数据集及数据源类型 ) async def main(): client = AgenticRL() result = await client.run( job_name="my-opd-job", # 任务名（控制台展示用） model="qwen3.5-9b", # 学生模型（基座模型，被蒸馏对象） teacher_model="your-teacher-model-id", # 教师模型，OPD 唯一新增参数；传值即触发蒸馏 training_datasets=[TrainingDataset( data_source_type=DataSourceType.FILE_ID, # 以文件方式上传训练数据 file_name="./data/train.jsonl")], validation_datasets=[ValidationDataset( data_source_type=DataSourceType.FILE_ID, # 验证集同样以文件方式上传 file_name="./data/validation.jsonl")], functions=None, # 纯蒸馏：不注册任何函数；Agentic OPD 在此传 Rollout/Reward 函数 hyper_parameters={ "n_epochs": 1, "n_rollouts": 8, "learning_rate": 1e-6, # 蒸馏是微调模型已有能力，学习率起点 1e-6 "eval_steps": 1, "ppo_mini_batch_size": 8, "lr_scheduler_type": "cosine", # kl_loss_coef OPD 未启用，见训练配置 # opd_teacher_* 部署参数见训练配置 }, resources={ "charge_type": "mtu_postpaid", # MTU 后付费计费 "mtu_spec_code": "MTU4", # 训练单元规格 "mtu_capacity": 24, # 容量（并发资源数） }, ) # HTTP 200 返回 job_id（查询训练状态的唯一标识）；否则打印原始响应便于排查 print(result.output.job_id if result.status_code == 200 else result) asyncio.run(main()) # 启动异步主函数 ``` Can also via config file Or CLI (`dashscope rl run`) Submit; three methods with field-by-field explanation see[Training Configuration](opd-training-config.md)。 ### Getting Started Tips - First choose an example: for plain text output tasks start from the Model OPD example; when output depends on API-level tool call Or multi-turn/custom interaction, start from the Agentic OPD example. - Use streamlined data first: dozens of items + `n_epochs=1` to verify the pipeline before switching to the full dataset, avoiding consuming MTU on the full dataset only to discover config errors later. - Keep hyperparameters Default：`learning_rate` as starting point `1e-6`(distillation fine-tunes the model's existing capabilities rather than rebuilding, too large a learning rate easily breaks output Format),`lr_scheduler_type` use `cosine`; when tuning, only change one variable at a time, at least train `eval_steps × 3` steps before judging the trend. - FAILED check logs first: Task Failed → log tab to view the error at the end, Or SDK Calling `AgenticRL.logs(job_id="ft-xxx", lines=100)`。 ## Post-training Process ### Observation and Evaluation `run()` returned `job_id` is the unique identifier for querying training status.[Model Fine-tuning Console](https://bailian.console.aliyun.com/cn-beijing/model/tuning)Enter the task detail page, containing four tabs: metrics, Trajectory, Output, and logs. Observation focus: - `trace/distillation/valid_token_rate` close to 1.0 (teacher model signal complete). - `actor/distillation/loss` decreases Or stabilizes (student model converges toward teacher model). - When Enable Reward, additionally watch `validation/data/reward/mean@1` and `trace/reward_metrics` rise relative to baseline across all dimensions. Post-training evaluation requires preserving the pre-training baseline and maintaining consistent evaluation conditions (same eval set and decoding Parameter). Complete metric meanings, Trajectory viewing, acceptance criteria, and troubleshooting decision tree see[Observability Configuration](opd-observable-config.md)。 ### Publish and Deploy After training completes, the last Checkpoint is automatically published to[My Models](https://bailian.console.aliyun.com/cn-beijing/model/custom). To publish an intermediate Checkpoint, Go to [Model Fine-tuning Console](https://bailian.console.aliyun.com/cn-beijing/model/tuning)Output page for manual operation. Published models can be deployed on the My Models page, and after deployment completes, can be called via API. ## Applicable Scenario Applicable Scenario expands progressively by capability complexity: pure distillation is the Basic capability, Reward and Custom Rollout are advanced capabilities layered on top on demand (introducing extra components and more complex training/generation pipelines). - Pure distillation (Basic): only teacher model + data, no function components written → [Model OPD Development](opd-model-development-guide.md) - +Reward (advanced): add a scoring function, layering task reward on top of teacher model's per-token guidance (score by standard answer, score by multi-dimensional metrics) → [Model OPD Development](opd-model-development-guide.md) - +Custom Rollout (advanced, Agentic OPD): when output depends on tool Calling/multi-turn interaction, customize the generation process; can be layered with Reward, or used alone → [Agentic OPD Development](opd-agentic-development-guide.md) Two advanced capabilities can be enabled independently and combined freely; selection starts with the generation path: single-turn generation suffices → the first two levels (Model OPD); requires structured mechanisms (e.g., native tool call) or custom interaction → the third level (Agentic OPD). ## Next Steps - [Model OPD Development](opd-model-development-guide.md): complete details of function development (Rollout/Reward functions, Tracing integration). - [Agentic OPD Development](opd-agentic-development-guide.md): Custom Rollout (passing in the tools parameter, structured output), Agentic Reward development. - [Training Configuration](opd-training-config.md): Submission Method, field-by-field descriptions, hyperparameters and Environment Variable. - [Observability Configuration](opd-observable-config.md): metric meanings, Trajectory viewing, acceptance criteria and troubleshooting decision tree. ## FAQ ### When to choose OPD over SFT Need per-token error correction, student model learns from its own generations → OPD; only have fixed annotations, budget-sensitive and no online correction required → SFT. ### Can training effectiveness be judged without reward Yes. Pure distillation relies on `valid_token_rate` approaching 1.0 and `loss` declining and stabilizing to determine whether the teacher model's signal has been absorbed; the final effect is judged by a post-training independent evaluation (retain the baseline, compare under the same criteria). If you need to optimize online during training according to task objectives, then add Reward (an advanced usage of Model OPD), at the cost of needing to provide a computable Reward function.