Online Policy Distillation Training Configuration

OPD requires configuring a teacher model and hyperparameters when submitting a training job. This document covers three submission methods, teacher model configuration, hyperparameter values, tuning decisions, job management, and common troubleshooting.

Submit a Training Job

Three submission methods, with increasing flexibility. Configuration priority:__init__()(SDK init function) Default Value < YAML < run() runtime parameters (the latter overrides the former). Pass in teacher_model to trigger distillation. Pass AgenticRL().run(...) to complete function registration, data upload, and job submission in one step; pass an additional teacher_model to trigger distillation.

from dashscope.finetune.agentic_rl import AgenticRL          # OPD 训练客户端
from dashscope.finetune.reinforcement import (               # 组件与数据类型
    RewardFunctionComponent, FunctionComponentModel,
    FunctionComponentRuntime, TrainingDataset,
    ValidationDataset, DataSourceType,
)

client = AgenticRL()                                        # 初始化训练客户端
result = await client.run(                                  # 异步提交：注册函数→上传数据→提交任务
    job_name="my-opd-job",
    model="qwen3.5-9b",                        # 学生模型（训练对象，被蒸馏的学生）
    teacher_model="qwen3.5-397b-a17b",         # 教师模型，OPD 唯一新增参数；传入即触发蒸馏
    training_datasets=[TrainingDataset(        # 训练集：FILE_ID 方式由 SDK 自动上传并换 file_id
        data_source_type=DataSourceType.FILE_ID,
        file_name="./data/train.jsonl")],
    validation_datasets=[ValidationDataset(    # 验证集：用于训练中评估，指标驱动调参
        data_source_type=DataSourceType.FILE_ID,
        file_name="./data/validation.jsonl")],
    functions=[
        # 不传 Rollout 组件 → Model OPD（原生 Rollout）；传 Rollout 组件 → Agentic OPD（自定义 Rollout，支持多轮/自定义交互）
        RewardFunctionComponent(               # 仅传 Reward 组件：在原生 Rollout + 教师蒸馏基础上叠加任务奖励
            name="reward-1", weight=1.0, timeout=120,   # weight=奖励权重，timeout=单次打分超时秒数
            fcmodel=FunctionComponentModel(
                zipdir=".",  # 工作区根目录（含 functions/），SDK 打包此目录上传至 FC
                classpath="functions.reward.reward.MyRewardProcessor"),  # 入口类全路径（示例类名，实际由您自定义）
            runtime=FunctionComponentRuntime(
                cpu=2, memory_size=4096, disk_size=512,            # 函数实例 CPU/内存/磁盘
                concurrency=20, capacity=10, min_capacity=10, max_capacity=30)),  # 并发与弹性容量
    ],
    # 纯蒸馏场景：改传 functions=None（而非空列表 []），不注册任何函数，reward 派生指标不上报
    hyper_parameters={                         # 训练超参，完整字段见下方 §训练超参数
        "algorithm": "gspo", "learning_rate": 1e-6, "lr_scheduler_type": "cosine",
        "n_epochs": 1, "n_rollouts": 8, "batch_size": 128,
    },
    resources={"charge_type": "mtu_postpaid", "mtu_spec_code": "MTU4", "mtu_capacity": 24},  # 计费资源：后付费/规格/容量
)

resources Configure training billing resources:charge_type: mtu_postpaid Indicates pay-as-you-go billing by MTU (Model Studio compute Type unit),mtu_spec_code specify the Type (e.g. MTU4), mtu_capacity specify the Capacity.

functions The Parameter determines the run mode (pure distillation / Model OPD / Agentic OPD), and teacher_model is unrelated ——teacher_model only triggers distillation. For mode selection, see Overview of applicable scenarios. Here only the SDK-layer value mapping is listed; the distinguishing criterion is whether a Rollout Component is passed (determines using native Rollout or Custom Rollout):

Model OPD: do not pass a Rollout Component → use native Rollout, including pure distillation (functions=None) and add Reward (only pass Reward Component to overlay task reward) two tiers
Agentic OPD (None Reward): only pass Rollout Component → Custom Rollout (supports multi-turn dialogue, custom interaction, structured tool call) + teacher distillation
Agentic OPD (with Reward): simultaneously pass Rollout + Reward → Custom Rollout + teacher distillation + Reward
Pass None(rather than empty list []) → pure distillation, reward derived metrics not reported, distillation loss-type metrics and teacher health-type metrics still reported normally

YAML Configure

Suitable for multiple experiments or managing multiple groups of parameters, via teacher_model Field specifies the teacher model.

## config.yaml（简化示意）
name: "my-opd-job"
model:
  name: "qwen3.5-9b"                           # 学生模型（训练对象）
teacher_model: "qwen3.5-397b-a17b"             # 教师模型（OPD 新增），传入即触发蒸馏

datasets:
  - type: "training"                           # 训练集
    data_source_type: "file_id"                # file_id 方式：SDK 自动上传并换 file_id
    file_name: "./data/train.jsonl"
  - type: "validation"                         # 验证集，训练中评估用
    data_source_type: "file_id"
    file_name: "./data/validation.jsonl"

functions:                                     # 不传 Rollout 组件即 Model OPD；纯蒸馏写 functions: null
  - type: "reward"                             # 仅传 Reward：原生 Rollout + 教师蒸馏 + 任务奖励
    name: "reward-1"
    weight: 1.0                                # 奖励权重，多 reward 时加权求和
    timeout: 120                               # 单次打分超时秒数
    fcmodel:
      classpath: "functions.reward.reward.MyRewardProcessor"  # 入口类全路径（示例类名，实际由您自定义）
    runtime:
      cpu: 2                                   # 函数实例 CPU 核数
      memory_size: 4096                        # 内存 MB
      concurrency: 20                          # 单实例并发打分数
      capacity: 10                             # 当前实例数
      min_capacity: 10                         # 最小实例数（保底）
      max_capacity: 30                         # 最大实例数（弹性上限）

training:
  type: "pg_opd"                               # OPD 训练类型；传 teacher_model 后自动置此值，无需手填
  hyper_parameters:
    algorithm: "gspo"                          # 训练算法
    learning_rate: 1e-6                         # 学习率，OPD 起点低于 SFT
    lr_scheduler_type: "cosine"                 # 学习率调度器
    n_epochs: 1                                # 训练轮次
    # ... 完整字段见下方 §训练超参数
  resources:
    charge_type: "mtu_postpaid"                # 后付费计费
    mtu_spec_code: "MTU4"                       # 算力规格
    mtu_capacity: 24                           # 容量

Load and Execute:

client = AgenticRL()                                            # 初始化训练客户端
client.init(config_path="config.yaml", name="my-opd-from-yaml") # 加载 YAML 配置，name 覆盖 YAML 中的 job_name
result = await client.run()                                     # 异步提交任务；run() 参数覆盖 init() 与 YAML 默认值

CLI Command

Pass dashscope rl run Submit,--teacher-model overrides in YAML teacher_model.

dashscope rl run -c config.yaml --teacher-model qwen3.5-397b-a17b
# -c 指定 YAML 配置文件路径
# --teacher-model 指定教师模型，覆盖 YAML 中的 teacher_model 字段；传入即启用 OPD

--teacher-modelhelp text (original): Enable OPD with this teacher model (overrides YAML); Rollout and Reward are optional. Meaning: Enable OPD and specify the teacher model (overrides YAML); Rollout and Reward are Optional. Common CLI Commands:

Actions	Command	Description
Submit Training	`dashscope rl run -c config.yaml --teacher-model "teacher-model" [--verbose\|-o json]`	Read YAML configuration to submit; --verbose human-readable, -o json machine-readable
Upload Data	`dashscope rl upload_data --training-files "file-path" --validation-files "file-path" -o json`	Only upload data, return File ID

Example submission script submit_job.py (reference implementation, classpath is example class name)

Provided by the example directory submit_job.py Attach hyperparameters as command-line arguments, adjustable as needed.--no-reward is a convenience switch on the example side (not an SDK parameter), equivalent to the SDK layer functions=None. zipdir Recommend explicitly specifying to avoid entry-class localization failure. Key snippets are as follows:

import asyncio
from dashscope.finetune.agentic_rl import AgenticRL          # OPD 训练客户端
from dashscope.finetune.reinforcement import (                # 组件与数据类型
    RewardFunctionComponent, FunctionComponentModel,
    FunctionComponentRuntime, TrainingDataset,
    ValidationDataset, DataSourceType,
)

async def submit(args):
    client = AgenticRL()                                    # 初始化训练客户端
    # 纯蒸馏时传 None 而非空列表：None 使控制面（调度层）执行原生 Rollout，[] 不一定等价
    # --no-reward 是示例侧便利开关（非 SDK 参数），等价于 SDK 层 functions=None
    function_components = (
        None if args.no_reward                              # 纯蒸馏：不注册任何函数
        else [RewardFunctionComponent(                      # 加 Reward：原生 Rollout + 教师蒸馏 + 任务奖励
            name="reward-1", weight=1.0, timeout=120,       # weight=奖励权重，timeout=单次打分超时秒数
            fcmodel=FunctionComponentModel(
                zipdir=str(args.workspace_dir.resolve()),   # 工作区根目录，显式指定避免入口类定位失败
                classpath="functions.reward.reward.MyRewardProcessor"),  # 入口类全路径（示例类名，实际由您自定义）
            runtime=FunctionComponentRuntime(**build_runtime()))]        # 并发与弹性容量由 build_runtime() 构造
    )
    result = await client.run(                              # 异步提交：注册→上传→提交
        job_name=args.job_name,
        model=args.student_model,          # 学生模型（训练对象）
        teacher_model=args.teacher_model,  # 教师模型，传入即触发蒸馏
        training_datasets=[...],           # 训练集（示例省略构造）
        validation_datasets=[...],         # 验证集（示例省略构造）
        functions=function_components,     # None=纯蒸馏；[Reward...]=Model OPD；含 Rollout=Agentic OPD
        hyper_parameters={...},            # 训练超参（示例省略，见下方 §训练超参数）
    )
    return result.output.job_id            # 返回 job_id，供后续查询/取消/日志使用

Configure Teacher Model

teacher_model is the submission parameter that triggers distillation. Passing it triggers distillation; not passing it does not. After it is passed, the SDK internal field training_type is automatically set to pg_opd (strategy distillation training type identifier, corresponding to that in YAML training.type), no need to and cannot manually specify the training type. Behavior Conclusion:

Pass teacher_model forces training_type=pg_opd, no need to and cannot manually specify the training type
Write in YAML training.type: pg_opd but not passed teacher_model when, the control plane reports an error error 3006(teacher model missing)
hyper_parameters Undeclared keys will not be recognized, and will neither report errors nor take effect

Three hard constraints

Teacher model configuration must simultaneously satisfy the following three constraints; failure to meet any one will cause distillation to fail or the submission to be rejected.

Constraint	Description	Behavior when violated
Must be a Model Studio model ID	Only accepts Model Studio-registered model IDs, not HuggingFace repository names or weight paths	`400 InvalidParameter: teacher_model must be a valid model ID`
Teacher-student same model family	The teacher and student model tokenizers must be alignable (tokenizer alignment: consistent vocabulary, token ids correspond one-to-one, e.g., both belong to the Qwen3.5 series)	When vocabularies are not aligned, token-level distributions cannot correspond position by position, and the distillation signal fails
Teacher model stronger than student model	Distillation transfers LLM capabilities to a smaller model	No distillation gain when teacher is not stronger than student

Training Hyperparameters

hyper_parameters field names and value ranges are consistent, grouped into 4 categories: Algorithm (algorithm; kl_loss_coef not enabled in OPD) / Optimize (learning_rate, lr_scheduler_type, lr_warmup_steps, n_epochs) / Data throughput (batch_size, n_rollouts, ppo_mini_batch_size, max_length, max_prompt_length) / Evaluation & save (eval_steps, save_steps, save_strategy, save_total_limit).

Required parameters comprehensive table

The comprehensive table below merges field definitions, value ranges, and recommended starting points by model scale × data phase; you can launch training directly by adopting the starting point column, then iterate per §tuning decision table.

OPD learning rate starting point is lower (1e-6): distillation fine-tunes the student model's existing capabilities rather than rebuilding them; an overly large learning rate tends to break the output format first.

Parameter	Introduction	Value Range	Small Model Quick Validation	Formal Training	Large Model
algorithm	Training Algorithm	gspo	gspo	gspo	gspo
kl_loss_coef	KL Loss Coefficient (OPD Not Enabled)	[0, 10]	—	—	—
learning_rate	Learning Rate	[0, 1]	1e-6	3e-6 ~ 5e-6	2e-6 ~ 3e-6
lr_scheduler_type	Scheduler	cosine / constant	cosine	cosine	cosine
lr_warmup_steps	Warmup Steps (-1 = Auto; set 0 for small-data smoke test to avoid exceeding total steps)	>= -1	0	-1	-1
n_epochs	Training Epochs	[1, 200]	1	2 ~ 3	1 ~ 2
batch_size	Samples per step	[1, 100000]	128	128 ~ 256	256 ~ 512
n_rollouts	Samples per prompt	[1, 1024]	8	8	8 ~ 16
ppo_mini_batch_size	PPO mini-batch	[1, 100000]	8	16 ~ 32	32 ~ 64
max_length	Max tokens for prompt + resp	[1, 131072]	8192	8192 ~ 16384	8192 ~ 16384
max_prompt_length	Max tokens for prompt	[1, 131072]	2048	2048 ~ 4096	2048 ~ 4096
eval_steps	Eval interval (Recommendation: total steps / 10~20)	[1, 999999]	1	Total steps / 10~20	Total steps / 10~20
save_steps	Save interval (Recommendation: eval_steps × 2~3)	[1, 999999]	1	eval_steps × 2~3	eval_steps × 2~3
save_strategy	Save strategy	steps / epoch	steps	steps	steps
save_total_limit	Max ckpt retained	[1, 200]	2	3	2 ~ 3

Model Scale Boundary: Student model parameter count ≤14B counts as small model, ≥30B counts as large model (e.g., Qwen3.5-9b is small, 397b-A17b is large). The "small model quick validation" column in the table applies to pipeline smoke testing and small-scale training, the "large model" column applies to formal large-scale training; for cases in between, take the union of both columns and fine-tune based on actual convergence. Three parameters that must be adjusted together(determines which must be linked during tuning):

Sample Count: batch_size × n_rollouts = actual samples generated per step
mini-batch upper limit: ppo_mini_batch_size ≤ batch_size × n_rollouts
learning_rate determines distillation convergence speed—OPD has no kl_loss_coef constrains model deviation from initial policy (prevents output style drift), learning rate is the only parameter controlling convergence speed

Optional Parameters

The following parameters are used to control distributed training resource configuration, generally using default values is sufficient. The default values in the table are the values when not passed; actual values are subject to what is passed at submission.

Parameter	Description	Default Value	Value Range
n_gpus_rollout	Number of GPUs in the Rollout Phase	8	[1, 1024]
n_gpus_training	Number of GPUs in the Training Phase	8	[1, 1024]
nnodes_rollout	Number of Nodes in the Rollout Phase	1	[1, 256]
nnodes_training	Number of Nodes in the Training Phase	2	[1, 256]

Teacher Model Deployment Parameters and Disabled Items

Pay special attention to the following parameters; understanding them helps you avoid mistakenly tuning parameters that will not take effect.

algorithm: Default gspo(GSPO, group relative policy optimization). When reward is enabled, determines the implementation method of policy optimization; pure distillation (without registering reward) does not participate in loss calculation, generally no modification needed
Teacher model deployment resources: can explicitly configure opd_teacher_* parameter, if not passed, takes the Default Value. Example default values are as follows:

Teacher model is deployed via vLLM (Model Studio reasoning engine); KV cache (mechanism in reasoning GPU memory that caches computed tokens) capacity determines scoring throughput. Adjust according to teacher model scale and sequence length:

Long sequence: increase opd_teacher_max_model_len(along with opd_teacher_max_num_batched_tokens↑)
Tight GPU memory: Lower opd_teacher_gpu_memory_utilization

Other parameters generally keep the Default.

Parameter	demo Default Value
opd_teacher_nnodes	1
opd_teacher_n_gpus_per_node	8
opd_teacher_infer_tp	8
opd_teacher_gpu_memory_utilization	0.75
opd_teacher_max_model_len	34816
opd_teacher_max_num_batched_tokens	4096
opd_teacher_max_num_seqs	2
opd_teacher_num_replicas	1

kl_loss_coef: OPD does not consume this parameter—distillation does not constrain policy divergence via KL divergence (a metric measuring the difference between two probability distributions), and convergence speed is controlled by learning_rate alone. The "—" marked in the required parameter table indicates OPD does not consume it. You can keep the Default or leave it empty; no need to zero it out or Delete it.

Parameter Tuning and Pitfall Avoidance

Tuning Decision Table

When metrics are abnormal, refer to this table to select the preferred adjustment item and magnitude; for complete metric meanings and troubleshooting decision tree, seeOnline Policy Distillation Observable Configuration.

Symptom	Primary Adjustment	Adjustment Range	Fallback
distillation/loss not decreasing	learning_rate ↑	× 1.5~2	Check valid_token_rate + Reward function
distillation/loss exploding	learning_rate ↓	× 0.5	Check teacher model health metrics
valid_token_rate < 1	Submit a ticket (teacher model side)	—	Confirm teacher-student same model family
Overfitting (Validation Set no longer improving)	n_epochs ↓	-1~2	save_strategy=steps early stop
Reward signal not taking effect (success_rate=0)	Check FC calls and Reward function	—	Reproduce scoring with local test_functions
Rollout slow	n_rollouts ↓ / max_length tighten	—	Add concurrency (FunctionComponentRuntime.concurrency/capacity, see Development)
OOM	ppo_mini_batch_size ↓	× 0.5	max_length ↓ / tune parallel parameters
High truncation rate	max_length ↑	+ 2048	Check max_prompt_length

Tuning Operation Recommendations

Check first valid_token_rate: When this metric is below 1, the teacher model scoring has gaps, and the explanatory power of other metrics decreases. Investigate this first before tuning hyperparameters.
Change only one variable at a time: Tuning both learning_rate and batch_size simultaneously makes attribution impossible.
Train for eval_steps × 3 steps before judging the trend: Distillation curves have less noise, but trends are hard to judge over short periods. Wait at least 3 evaluation intervals before judging the direction.

Anti-patterns

The following configurations have been proven in practice to cause training failure or resource waste and should be avoided.

Anti-pattern	Consequence
Execute OPD with SFT's lr starting point (5e-5)	Training diverges — distillation is sensitive to lr, one order of magnitude lower than SFT
Set n_epochs to 10+ expecting "thorough training"	Deepens overfitting to the teacher model's style, validation set collapses
Use HuggingFace repository name as teacher_model	400 InvalidParameter: teacher_model must be a valid model ID
Teacher model capability weaker than student model	No distillation gain, wasted compute
max_length=32K for safety	teacher vLLM reserves KV cache according to opd_teacher_max_model_len, and max_num_seqs is compressed (demo only 2), reducing scoring throughput; short sequences will not cause OOM. Set based on actual longest sequence ×1.2~1.5; 32K is only needed when sequences actually reach 32K
pass an empty list [] as pure distillation	the control plane may not handle [] and None the same way, possibly unable to execute native rollout; pure distillation must pass None

Task Management

This section provides commands for querying, canceling, deleting, and viewing logs of training tasks, as well as experiment management and continual training methods.

SDK / CLI Command Comparison

Task management commands (query, cancel, delete, logs) are as follows; pass an extra parameter when submitting teacher_model:

Actions	SDK	CLI
Query Status	`AgenticRL.get(job_id="ft-xxx")`	`dashscope rl get "ft-xxx" -o json`
List Tasks	`AgenticRL.list()`	`dashscope rl list --page 1 --size 10 -o json`
Cancel Task	`AgenticRL.cancel(job_id="ft-xxx")`	`dashscope rl cancel "ft-xxx"`
Delete Task	`AgenticRL.delete(job_id="ft-xxx")`	CLI not supported, use SDK
View Logs	`AgenticRL.logs(job_id="ft-xxx", lines=100)`	`dashscope rl logs "ft-xxx" --offset 1 --lines 100`

Experiment Management

YAML Multiple-experiment mode:job_name identifies experiments, finetuned_output identifies output models; change these two each time to isolate experiments; YAML centralizes configuration management for Git versioning
Configuration priority:__init__() < YAML < run() Runtime parameters — for controlled experiments only change the outermost layer
Parameter sweep / CI scenarios (advanced usage, general users can skip): when the function is stable and only hyperparameters change, you can functions pass at the Component level entity_id(such as RewardFunctionComponent(..., entity_id="rw-xxx")) reuse registered components, saving repeated registration and upload. Entity ID (ro- represents Rollout, rw- represents Reward) is taken from the return value of the first registration, Field Name entity_id, can be put into CI configuration for long-term reuse:

result = await client.run(...)                    # 首次提交：注册函数组件并上传代码/数据
entity_id = result.output.entity_id              # 从首次注册返回值取 entity_id，ro- 表示 Rollout、rw- 表示 Reward
# 后续扫参/CI 只换超参时，在组件级传 entity_id="rw-xxx" 复用已注册组件，省去重复注册与上传

Continued Training and Checkpoint

Checkpoint retention and selection:

save_strategy=steps + save_steps=100 + save_total_limit=3: Keep up to 3 piece, automatically clean the oldest when exceeded
Which Checkpoint to select: When reward is enabled, look at validation/data/reward/mean@1 select the best point (in pure distillation without reward enabled, this metric is not reported; based on actor/distillation/loss trend stabilized point selection), not necessarily the last one
Keep 3~5 piece to balance space and rollback needs

Continue training based on published Checkpoint:

Continue training = use the model produced last time as model input parameter and submit again; hyperparameters can be fine-tuned (lr is usually reduced)
Applicable: Data increment / Reward adjustment / long-chain gradual convergence
Note: The base model version must be consistent; cross-base-model cannot directly continue training
In a multi-phase training pipeline, each phase's output can serve as the next phase's model input, progressing level by level

Next Steps

Review principles and scenario selection → Online Policy Distillation Training Overview
Model OPD Development (Pure Distillation/Reward) → Model OPD Development
Agentic OPD Development (Custom Rollout) → Agentic OPD Development
Training Metrics Quick Reference, Effect Evaluation and Troubleshooting → Online Policy Distillation Observable Configuration

FAQ

Show teacher_model error 400 InvalidParameter

Filled in a HuggingFace repository name or weight path. Use the Model Studio model ID instead (e.g. qwen3.5-397b-a17b), and ensure the model has been activated in Model Studio.

Show SDK Version and Environment Variables

Installation is based on the wheel bundled in the examples directory (currently 1.27.5), and does not rely on the PyPI version number to determine whether OPD is supported.The old SDK has two interception points (distinguished by timing):

Pre-submission interception: example submit_job.py validates whether its signature accepts the AgenticRL.run(...) parameter before calling teacher_model — the old SDK signature does not include this parameter, so it errors before submission
Submission-time interception (old SDK troubleshooting): earlier versions write training_type as pg-opd (hyphen), while the control plane only accepts pg_opd (underscore), so submission is rejected

FC_LAYER_USED only recognizes true/1/yes; on is treated as false. Recommendation: only write lowercase true / false to avoid ambiguity from values like on.

Show Endpoint and URL Configuration

Two sets of endpoints: DASHSCOPE_HTTP_BASE_URL (control plane, /api/v1 submit training) and MODEL_BASE_URL (reasoning, /compatible-mode/v1 offline evaluation) are two sets of endpoints. The API Key must match the corresponding endpoint; a mismatch will return InvalidApiKey / HTTP 401.DASHSCOPE_HTTP_BASE_URL. If there are trailing spaces or / at the end, an illegal URL will be formed. Please ensure the end is clean.

​Submit a Training Job

​Recommend: one step done (run)

​YAML Configure

​CLI Command

​Configure Teacher Model

​Three hard constraints

​Training Hyperparameters

​Required parameters comprehensive table

​Optional Parameters

​Teacher Model Deployment Parameters and Disabled Items

​Parameter Tuning and Pitfall Avoidance

​Tuning Decision Table

​Tuning Operation Recommendations

​Anti-patterns

​Task Management

​SDK / CLI Command Comparison

​Experiment Management

​Continued Training and Checkpoint

​Next Steps

​FAQ

Submit a Training Job

Recommend: one step done (run)

YAML Configure

CLI Command

Configure Teacher Model

Three hard constraints

Training Hyperparameters

Required parameters comprehensive table

Optional Parameters

Teacher Model Deployment Parameters and Disabled Items

Parameter Tuning and Pitfall Avoidance

Tuning Decision Table

Tuning Operation Recommendations

Anti-patterns

Task Management

SDK / CLI Command Comparison

Experiment Management

Continued Training and Checkpoint

Next Steps

FAQ