Online Policy Distillation Training Configuration

> ## Documentation Index > Fetch the complete documentation index at: https://docs.bailian.console.aliyun.com/llms.txt > Use this file to discover all available pages before exploring further. # Online Policy Distillation Training Configuration > Model Studio online policy distillation training configuration: submission methods, teacher model configuration, hyperparameters, tuning pitfalls, task management, and FAQ OPD requires configuring a teacher model and hyperparameters when submitting a training job. This document covers three submission methods, teacher model configuration, hyperparameter values, tuning decisions, job management, and common troubleshooting. ## Submit a Training Job Three submission methods, with increasing flexibility. Configuration priority:`__init__()`(SDK init function) Default Value \< YAML \< `run()` runtime parameters (the latter overrides the former). Pass in `teacher_model` to trigger distillation. ### Recommend: one step done (run) Pass `AgenticRL().run(...)` to complete function registration, data upload, and job submission in one step; pass an additional `teacher_model` to trigger distillation. ```plaintext expandable from dashscope.finetune.agentic_rl import AgenticRL # OPD 训练客户端 from dashscope.finetune.reinforcement import ( # 组件与数据类型 RewardFunctionComponent, FunctionComponentModel, FunctionComponentRuntime, TrainingDataset, ValidationDataset, DataSourceType, ) client = AgenticRL() # 初始化训练客户端 result = await client.run( # 异步提交：注册函数→上传数据→提交任务 job_name="my-opd-job", model="qwen3.5-9b", # 学生模型（训练对象，被蒸馏的学生） teacher_model="qwen3.5-397b-a17b", # 教师模型，OPD 唯一新增参数；传入即触发蒸馏 training_datasets=[TrainingDataset( # 训练集：FILE_ID 方式由 SDK 自动上传并换 file_id data_source_type=DataSourceType.FILE_ID, file_name="./data/train.jsonl")], validation_datasets=[ValidationDataset( # 验证集：用于训练中评估，指标驱动调参 data_source_type=DataSourceType.FILE_ID, file_name="./data/validation.jsonl")], functions=[ # 不传 Rollout 组件 → Model OPD（原生 Rollout）；传 Rollout 组件 → Agentic OPD（自定义 Rollout，支持多轮/自定义交互） RewardFunctionComponent( # 仅传 Reward 组件：在原生 Rollout + 教师蒸馏基础上叠加任务奖励 name="reward-1", weight=1.0, timeout=120, # weight=奖励权重，timeout=单次打分超时秒数 fcmodel=FunctionComponentModel( zipdir=".", # 工作区根目录（含 functions/），SDK 打包此目录上传至 FC classpath="functions.reward.reward.MyRewardProcessor"), # 入口类全路径（示例类名，实际由您自定义） runtime=FunctionComponentRuntime( cpu=2, memory_size=4096, disk_size=512, # 函数实例 CPU/内存/磁盘 concurrency=20, capacity=10, min_capacity=10, max_capacity=30)), # 并发与弹性容量 ], # 纯蒸馏场景：改传 functions=None（而非空列表 []），不注册任何函数，reward 派生指标不上报 hyper_parameters={ # 训练超参，完整字段见下方 §训练超参数 "algorithm": "gspo", "learning_rate": 1e-6, "lr_scheduler_type": "cosine", "n_epochs": 1, "n_rollouts": 8, "batch_size": 128, }, resources={"charge_type": "mtu_postpaid", "mtu_spec_code": "MTU4", "mtu_capacity": 24}, # 计费资源：后付费/规格/容量 ) ``` `resources` Configure training billing resources:`charge_type: mtu_postpaid` Indicates pay-as-you-go billing by MTU (Model Studio compute Type unit),`mtu_spec_code` specify the Type (e.g. `MTU4`), `mtu_capacity` specify the Capacity. `functions` The Parameter determines the run mode (pure distillation / Model OPD / Agentic OPD), and `teacher_model` is unrelated ——`teacher_model` only triggers distillation. For mode selection, see [Overview of applicable scenarios](opd-training-overview.md#opd-scenarios). Here only the SDK-layer value mapping is listed; the distinguishing criterion is whether a Rollout Component is passed (determines using native Rollout or Custom Rollout): - **Model OPD**: do not pass a Rollout Component → use native Rollout, including pure distillation (`functions=None`) and add Reward (only pass Reward Component to overlay task reward) two tiers - **Agentic OPD (None Reward)**: only pass Rollout Component → Custom Rollout (supports multi-turn dialogue, custom interaction, structured tool call) + teacher distillation - **Agentic OPD (with Reward)**: simultaneously pass Rollout + Reward → Custom Rollout + teacher distillation + Reward - Pass `None`(rather than empty list `[]`) → pure distillation, reward derived metrics not reported, distillation loss-type metrics and teacher health-type metrics still reported normally ### YAML Configure Suitable for multiple experiments or managing multiple groups of parameters, via `teacher_model` Field specifies the teacher model. ```plaintext expandable ## config.yaml（简化示意） name: "my-opd-job" model: name: "qwen3.5-9b" # 学生模型（训练对象） teacher_model: "qwen3.5-397b-a17b" # 教师模型（OPD 新增），传入即触发蒸馏 datasets: - type: "training" # 训练集 data_source_type: "file_id" # file_id 方式：SDK 自动上传并换 file_id file_name: "./data/train.jsonl" - type: "validation" # 验证集，训练中评估用 data_source_type: "file_id" file_name: "./data/validation.jsonl" functions: # 不传 Rollout 组件即 Model OPD；纯蒸馏写 functions: null - type: "reward" # 仅传 Reward：原生 Rollout + 教师蒸馏 + 任务奖励 name: "reward-1" weight: 1.0 # 奖励权重，多 reward 时加权求和 timeout: 120 # 单次打分超时秒数 fcmodel: classpath: "functions.reward.reward.MyRewardProcessor" # 入口类全路径（示例类名，实际由您自定义） runtime: cpu: 2 # 函数实例 CPU 核数 memory_size: 4096 # 内存 MB concurrency: 20 # 单实例并发打分数 capacity: 10 # 当前实例数 min_capacity: 10 # 最小实例数（保底） max_capacity: 30 # 最大实例数（弹性上限） training: type: "pg_opd" # OPD 训练类型；传 teacher_model 后自动置此值，无需手填 hyper_parameters: algorithm: "gspo" # 训练算法 learning_rate: 1e-6 # 学习率，OPD 起点低于 SFT lr_scheduler_type: "cosine" # 学习率调度器 n_epochs: 1 # 训练轮次 # ... 完整字段见下方 §训练超参数 resources: charge_type: "mtu_postpaid" # 后付费计费 mtu_spec_code: "MTU4" # 算力规格 mtu_capacity: 24 # 容量 ``` Load and Execute: ```plaintext client = AgenticRL() # 初始化训练客户端 client.init(config_path="config.yaml", name="my-opd-from-yaml") # 加载 YAML 配置，name 覆盖 YAML 中的 job_name result = await client.run() # 异步提交任务；run() 参数覆盖 init() 与 YAML 默认值 ``` ### CLI Command Pass `dashscope rl run` Submit,`--teacher-model` overrides in YAML `teacher_model`. ```plaintext dashscope rl run -c config.yaml --teacher-model qwen3.5-397b-a17b # -c 指定 YAML 配置文件路径 # --teacher-model 指定教师模型，覆盖 YAML 中的 teacher_model 字段；传入即启用 OPD ``` `--teacher-model`help text (original): Enable OPD with this teacher model (overrides YAML); Rollout and Reward are optional. Meaning: Enable OPD and specify the teacher model (overrides YAML); Rollout and Reward are Optional. Common CLI Commands:

Actions	Command	Description
Submit Training	`dashscope rl run -c config.yaml --teacher-model "teacher-model" \[--verbose\|-o json]`	Read YAML configuration to submit; --verbose human-readable, -o json machine-readable
Upload Data	`dashscope rl upload\_data --training-files "file-path" --validation-files "file-path" -o json`	Only upload data, return File ID

Provided by the example directory `submit_job.py` Attach hyperparameters as command-line arguments, adjustable as needed.`--no-reward` is a convenience switch on the example side (not an SDK parameter), equivalent to the SDK layer `functions=None`. `zipdir` Recommend explicitly specifying to avoid entry-class localization failure. Key snippets are as follows: ```plaintext expandable import asyncio from dashscope.finetune.agentic_rl import AgenticRL # OPD 训练客户端 from dashscope.finetune.reinforcement import ( # 组件与数据类型 RewardFunctionComponent, FunctionComponentModel, FunctionComponentRuntime, TrainingDataset, ValidationDataset, DataSourceType, ) async def submit(args): client = AgenticRL() # 初始化训练客户端 # 纯蒸馏时传 None 而非空列表：None 使控制面（调度层）执行原生 Rollout，[] 不一定等价 # --no-reward 是示例侧便利开关（非 SDK 参数），等价于 SDK 层 functions=None function_components = ( None if args.no_reward # 纯蒸馏：不注册任何函数 else [RewardFunctionComponent( # 加 Reward：原生 Rollout + 教师蒸馏 + 任务奖励 name="reward-1", weight=1.0, timeout=120, # weight=奖励权重，timeout=单次打分超时秒数 fcmodel=FunctionComponentModel( zipdir=str(args.workspace_dir.resolve()), # 工作区根目录，显式指定避免入口类定位失败 classpath="functions.reward.reward.MyRewardProcessor"), # 入口类全路径（示例类名，实际由您自定义） runtime=FunctionComponentRuntime(**build_runtime()))] # 并发与弹性容量由 build_runtime() 构造 ) result = await client.run( # 异步提交：注册→上传→提交 job_name=args.job_name, model=args.student_model, # 学生模型（训练对象） teacher_model=args.teacher_model, # 教师模型，传入即触发蒸馏 training_datasets=[...], # 训练集（示例省略构造） validation_datasets=[...], # 验证集（示例省略构造） functions=function_components, # None=纯蒸馏；[Reward...]=Model OPD；含 Rollout=Agentic OPD hyper_parameters={...}, # 训练超参（示例省略，见下方 §训练超参数） ) return result.output.job_id # 返回 job_id，供后续查询/取消/日志使用 ``` ## Configure Teacher Model `teacher_model` is the submission parameter that triggers distillation. Passing it triggers distillation; not passing it does not. After it is passed, the SDK internal field `training_type` is automatically set to `pg_opd` (strategy distillation training type identifier, corresponding to that in YAML `training.type`), no need to and cannot manually specify the training type. Behavior Conclusion: - Pass `teacher_model` forces `training_type=pg_opd`, no need to and cannot manually specify the training type - Write in YAML `training.type: pg_opd` but not passed `teacher_model` when, the control plane reports an error `error 3006`(teacher model missing) - `hyper_parameters` Undeclared keys will not be recognized, and will neither report errors nor take effect ### Three hard constraints Teacher model configuration must simultaneously satisfy the following three constraints; failure to meet any one will cause distillation to fail or the submission to be rejected.

Constraint	Description	Behavior when violated
Must be a Model Studio model ID	Only accepts Model Studio-registered model IDs, not HuggingFace repository names or weight paths	`400 InvalidParameter: teacher\_model must be a valid model ID`
Teacher-student same model family	The teacher and student model tokenizers must be alignable (tokenizer alignment: consistent vocabulary, token ids correspond one-to-one, e.g., both belong to the Qwen3.5 series)	When vocabularies are not aligned, token-level distributions cannot correspond position by position, and the distillation signal fails
Teacher model stronger than student model	Distillation transfers LLM capabilities to a smaller model	No distillation gain when teacher is not stronger than student

## Training Hyperparameters `hyper_parameters` field names and value ranges are consistent, grouped into 4 categories: Algorithm (algorithm; kl\_loss\_coef not enabled in OPD) / Optimize (learning\_rate, lr\_scheduler\_type, lr\_warmup\_steps, n\_epochs) / Data throughput (batch\_size, n\_rollouts, ppo\_mini\_batch\_size, max\_length, max\_prompt\_length) / Evaluation & save (eval\_steps, save\_steps, save\_strategy, save\_total\_limit). ### Required parameters comprehensive table The comprehensive table below merges field definitions, value ranges, and recommended starting points by model scale × data phase; you can launch training directly by adopting the starting point column, then iterate per §tuning decision table. OPD learning rate starting point is lower (1e-6): distillation fine-tunes the student model's existing capabilities rather than rebuilding them; an overly large learning rate tends to break the output format first.

Parameter	Introduction	Value Range	Small Model Quick Validation	Formal Training	Large Model
algorithm	Training Algorithm	gspo	gspo	gspo	gspo
kl\_loss\_coef	KL Loss Coefficient (OPD Not Enabled)	\[0, 10]	—	—	—
learning\_rate	Learning Rate	\[0, 1]	1e-6	3e-6 \~ 5e-6	2e-6 \~ 3e-6
lr\_scheduler\_type	Scheduler	cosine / constant	cosine	cosine	cosine
lr\_warmup\_steps	Warmup Steps (-1 = Auto; set 0 for small-data smoke test to avoid exceeding total steps)	>= -1	0	-1	-1
n\_epochs	Training Epochs	\[1, 200]	1	2 \~ 3	1 \~ 2
batch\_size	Samples per step	\[1, 100000]	128	128 \~ 256	256 \~ 512
n\_rollouts	Samples per prompt	\[1, 1024]	8	8	8 \~ 16
ppo\_mini\_batch\_size	PPO mini-batch	\[1, 100000]	8	16 \~ 32	32 \~ 64
max\_length	Max tokens for prompt + resp	\[1, 131072]	8192	8192 \~ 16384	8192 \~ 16384
max\_prompt\_length	Max tokens for prompt	\[1, 131072]	2048	2048 \~ 4096	2048 \~ 4096
eval\_steps	Eval interval (Recommendation: total steps / 10\~20)	\[1, 999999]	1	Total steps / 10\~20	Total steps / 10\~20
save\_steps	Save interval (Recommendation: eval\_steps × 2\~3)	\[1, 999999]	1	eval\_steps × 2\~3	eval\_steps × 2\~3
save\_strategy	Save strategy	steps / epoch	steps	steps	steps
save\_total\_limit	Max ckpt retained	\[1, 200]	2	3	2 \~ 3

Model Scale Boundary: Student model parameter count ≤14B counts as small model, ≥30B counts as large model (e.g., Qwen3.5-9b is small, 397b-A17b is large). The "small model quick validation" column in the table applies to pipeline smoke testing and small-scale training, the "large model" column applies to formal large-scale training; for cases in between, take the union of both columns and fine-tune based on actual convergence. Three parameters that must be adjusted together(determines which must be linked during tuning): - **Sample Count**: `batch_size × n_rollouts = actual samples generated per step` - **mini-batch upper limit**: `ppo_mini_batch_size ≤ batch_size × n_rollouts` - `learning_rate` determines distillation convergence speed—OPD has no `kl_loss_coef` constrains model deviation from initial policy (prevents output style drift), learning rate is the only parameter controlling convergence speed ### Optional Parameters The following parameters are used to control distributed training resource configuration, generally using default values is sufficient. The default values in the table are the values when not passed; actual values are subject to what is passed at submission.

Parameter	Description	Default Value	Value Range
n\_gpus\_rollout	Number of GPUs in the Rollout Phase	8	\[1, 1024]
n\_gpus\_training	Number of GPUs in the Training Phase	8	\[1, 1024]
nnodes\_rollout	Number of Nodes in the Rollout Phase	1	\[1, 256]
nnodes\_training	Number of Nodes in the Training Phase	2	\[1, 256]

### Teacher Model Deployment Parameters and Disabled Items Pay special attention to the following parameters; understanding them helps you avoid mistakenly tuning parameters that will not take effect. - algorithm: Default `gspo`(GSPO, group relative policy optimization). When reward is enabled, determines the implementation method of policy optimization; pure distillation (without registering reward) does not participate in loss calculation, generally no modification needed - Teacher model deployment resources: can explicitly configure `opd_teacher_*` parameter, if not passed, takes the Default Value. Example default values are as follows: Teacher model is deployed via vLLM (Model Studio reasoning engine); KV cache (mechanism in reasoning GPU memory that caches computed tokens) capacity determines scoring throughput. Adjust according to teacher model scale and sequence length: - **Long sequence**: increase `opd_teacher_max_model_len`(along with `opd_teacher_max_num_batched_tokens`↑) - **Tight GPU memory**: Lower `opd_teacher_gpu_memory_utilization` Other parameters generally keep the Default.

Parameter	demo Default Value
opd\_teacher\_nnodes	1
opd\_teacher\_n\_gpus\_per\_node	8
opd\_teacher\_infer\_tp	8
opd\_teacher\_gpu\_memory\_utilization	0.75
opd\_teacher\_max\_model\_len	34816
opd\_teacher\_max\_num\_batched\_tokens	4096
opd\_teacher\_max\_num\_seqs	2
opd\_teacher\_num\_replicas	1

- kl\_loss\_coef: OPD does not consume this parameter—distillation does not constrain policy divergence via KL divergence (a metric measuring the difference between two probability distributions), and convergence speed is controlled by `learning_rate` alone. The "—" marked in the required parameter table indicates OPD does not consume it. You can keep the Default or leave it empty; no need to zero it out or Delete it. ## Parameter Tuning and Pitfall Avoidance ### Tuning Decision Table When metrics are abnormal, refer to this table to select the preferred adjustment item and magnitude; for complete metric meanings and troubleshooting decision tree, see[Online Policy Distillation Observable Configuration](opd-observable-config.md).

Symptom	Primary Adjustment	Adjustment Range	Fallback
distillation/loss not decreasing	learning\_rate ↑	× 1.5\~2	Check valid\_token\_rate + Reward function
distillation/loss exploding	learning\_rate ↓	× 0.5	Check teacher model health metrics
valid\_token\_rate \< 1	Submit a ticket (teacher model side)	—	Confirm teacher-student same model family
Overfitting (Validation Set no longer improving)	n\_epochs ↓	-1\~2	save\_strategy=steps early stop
Reward signal not taking effect (success\_rate=0)	Check FC calls and Reward function	—	Reproduce scoring with local test\_functions
Rollout slow	n\_rollouts ↓ / max\_length tighten	—	Add concurrency (FunctionComponentRuntime.concurrency/capacity, see Development)
OOM	ppo\_mini\_batch\_size ↓	× 0.5	max\_length ↓ / tune parallel parameters
High truncation rate	max\_length ↑	+ 2048	Check max\_prompt\_length

### Tuning Operation Recommendations - Check first `valid_token_rate`: When this metric is below 1, the teacher model scoring has gaps, and the explanatory power of other metrics decreases. Investigate this first before tuning hyperparameters. - Change only one variable at a time: Tuning both `learning_rate` and `batch_size` simultaneously makes attribution impossible. - Train for `eval_steps × 3` steps before judging the trend: Distillation curves have less noise, but trends are hard to judge over short periods. Wait at least 3 evaluation intervals before judging the direction. ### Anti-patterns The following configurations have been proven in practice to cause training failure or resource waste and should be avoided.

Anti-pattern	Consequence
Execute OPD with SFT's lr starting point (5e-5)	Training diverges — distillation is sensitive to lr, one order of magnitude lower than SFT
Set n\_epochs to 10+ expecting "thorough training"	Deepens overfitting to the teacher model's style, validation set collapses
Use HuggingFace repository name as teacher\_model	400 InvalidParameter: teacher\_model must be a valid model ID
Teacher model capability weaker than student model	No distillation gain, wasted compute
max\_length=32K for safety	teacher vLLM reserves KV cache according to opd\_teacher\_max\_model\_len, and max\_num\_seqs is compressed (demo only 2), reducing scoring throughput; short sequences will not cause OOM. Set based on actual longest sequence ×1.2\~1.5; 32K is only needed when sequences actually reach 32K
pass an empty list \[] as pure distillation	the control plane may not handle \[] and None the same way, possibly unable to execute native rollout; pure distillation must pass None

## Task Management This section provides commands for querying, canceling, deleting, and viewing logs of training tasks, as well as experiment management and continual training methods. ### SDK / CLI Command Comparison Task management commands (query, cancel, delete, logs) are as follows; pass an extra parameter when submitting `teacher_model`:

Actions	SDK	CLI
Query Status	`AgenticRL.get(job\_id="ft-xxx")`	`dashscope rl get "ft-xxx" -o json`
List Tasks	`AgenticRL.list()`	`dashscope rl list --page 1 --size 10 -o json`
Cancel Task	`AgenticRL.cancel(job\_id="ft-xxx")`	`dashscope rl cancel "ft-xxx"`
Delete Task	`AgenticRL.delete(job\_id="ft-xxx")`	CLI not supported, use SDK
View Logs	`AgenticRL.logs(job\_id="ft-xxx", lines=100)`	`dashscope rl logs "ft-xxx" --offset 1 --lines 100`

### Experiment Management - YAML Multiple-experiment mode:`job_name` identifies experiments, `finetuned_output` identifies output models; change these two each time to isolate experiments; YAML centralizes configuration management for Git versioning - Configuration priority:`__init__()` \< YAML \< `run()` Runtime parameters — for controlled experiments only change the outermost layer - Parameter sweep / CI scenarios (advanced usage, general users can skip): when the function is stable and only hyperparameters change, you can `functions` pass at the Component level `entity_id`(such as `RewardFunctionComponent(..., entity_id="rw-xxx")`) reuse registered components, saving repeated registration and upload. Entity ID (`ro-` represents Rollout, `rw-` represents Reward) is taken from the return value of the first registration, Field Name `entity_id`, can be put into CI configuration for long-term reuse: ```plaintext result = await client.run(...) # 首次提交：注册函数组件并上传代码/数据 entity_id = result.output.entity_id # 从首次注册返回值取 entity_id，ro- 表示 Rollout、rw- 表示 Reward # 后续扫参/CI 只换超参时，在组件级传 entity_id="rw-xxx" 复用已注册组件，省去重复注册与上传 ``` ### Continued Training and Checkpoint Checkpoint retention and selection: - `save_strategy=steps` + `save_steps=100` + `save_total_limit=3`: Keep up to 3 piece, automatically clean the oldest when exceeded - Which Checkpoint to select: When reward is enabled, look at `validation/data/reward/mean@1` select the best point (in pure distillation without reward enabled, this metric is not reported; based on `actor/distillation/loss` trend stabilized point selection), not necessarily the last one - Keep 3\~5 piece to balance space and rollback needs Continue training based on published Checkpoint: - Continue training = use the model produced last time as `model` input parameter and submit again; hyperparameters can be fine-tuned (lr is usually reduced) - Applicable: Data increment / Reward adjustment / long-chain gradual convergence - Note: The base model version must be consistent; cross-base-model cannot directly continue training - In a multi-phase training pipeline, each phase's output can serve as the next phase's `model` input, progressing level by level ## Next Steps - Review principles and scenario selection → [Online Policy Distillation Training Overview](opd-training-overview.md) - Model OPD Development (Pure Distillation/Reward) → [Model OPD Development](opd-model-development-guide.md) - Agentic OPD Development (Custom Rollout) → [Agentic OPD Development](opd-agentic-development-guide.md) - Training Metrics Quick Reference, Effect Evaluation and Troubleshooting → [Online Policy Distillation Observable Configuration](opd-observable-config.md) ## FAQ Filled in a HuggingFace repository name or weight path. Use the Model Studio model ID instead (e.g. `qwen3.5-397b-a17b`), and ensure the model has been activated in Model Studio. Installation is based on the wheel bundled in the examples directory (currently 1.27.5), and does not rely on the PyPI version number to determine whether OPD is supported. The old SDK has two interception points (distinguished by timing): - Pre-submission interception: example `submit_job.py` validates whether its signature accepts the `AgenticRL.run(...)` parameter before calling `teacher_model` — the old SDK signature does not include this parameter, so it errors before submission - Submission-time interception (old SDK troubleshooting): earlier versions write `training_type` as `pg-opd` (hyphen), while the control plane only accepts `pg_opd` (underscore), so submission is rejected `FC_LAYER_USED` only recognizes `true`/`1`/`yes`; `on` is treated as `false`. Recommendation: only write lowercase `true` / `false` to avoid ambiguity from values like `on`. Two sets of endpoints: `DASHSCOPE_HTTP_BASE_URL` (control plane, `/api/v1` submit training) and `MODEL_BASE_URL` (reasoning, `/compatible-mode/v1` offline evaluation) are two sets of endpoints. The API Key must match the corresponding endpoint; a mismatch will return `InvalidApiKey` / HTTP 401. `DASHSCOPE_HTTP_BASE_URL`. If there are trailing spaces or `/` at the end, an illegal URL will be formed. Please ensure the end is clean.