Online Policy Distillation Training Overview

​

Full Process Overview

1. Environment preparation — Install SDK, Obtain API Key, complete authorization
2. Prepare data and code — JSONL data, pure distillation does not need to write functions
3. Submit Training Task — Pass teacher_model to trigger distillation, pure distillation with zero code
4. Training Status Observation — Check distillation loss, teacher model signal health, validation set metrics
5. Publish and Deploy Model — Final Checkpoint auto-published, called via API after deployment

​

Working Mechanism

训练数据（用户问题 + 参考答案）
            │
            ▼
学生模型生成：学生模型按当前策略生成候选输出
            │
            ▼
教师模型前向：教师模型对学生模型输出逐 token 给概率分布
            │
            ▼
蒸馏更新：学生模型最小化自身分布与教师模型分布差异（KL 散度）
            │
            ▼
（可选）Reward 评分：启用 Reward 时叠加任务奖励信号
            │
            ▼
循环迭代：用更新后的模型重新采样，逐轮向教师模型靠拢

​

Model and Training Unit

​

Quick Start

​

Environment Preparation

workspace/
├── dashscope-*.whl          # SDK wheel：示例附带的 dashscope 安装包，本机与远端均按此版本安装
├── functions/               # Rollout/Reward 函数代码目录；纯蒸馏 functions=None 可留空
├── data/                    # JSONL 训练/验证数据；每行一个样本（messages + rollout_extra）
├── requirements.txt         # 函数依赖清单；FC_LAYER_USED=true 时注册阶段据此构建 layer，跳过冷启动 pip
└── submit_job.py            # 提交脚本：调用 AgenticRL().run() 完成注册→上传→提交三步

# 安装示例目录附带的 dashscope SDK wheel 文件（版本随示例更新，当前 1.27.5）
# 该 wheel 同名相对路径需填入 FC_PYPI_LIB，供远端函数运行时安装
python -m pip install ./dashscope-*.whl

• DASHSCOPE_API_KEY(required): Authentication.
• FC_PYPI_LIB(required): specifies the SDK package installed in the function runtime, fill in the wheel relative path (resolved remotely, local absolute path cannot hit); pure distillation (functions=None) does not involve function runtime, this item can be skipped.
• FC_LAYER_USED(Optional, Default true): when true, builds layer based on requirements.txt during registration phase, skipping cold-start pip install; when false, installed by remote runtime cold-start.

export DASHSCOPE_API_KEY="sk-your-api-key"                       # 鉴权密钥，从百炼控制台 API-KEY 页获取（必需）
export FC_PYPI_LIB=./dashscope-1.27.5-py3-none-any.whl          # 远端函数运行时安装的 SDK 包，填 wheel 相对路径；纯蒸馏可不设（必需，纯蒸馏除外）
export FC_LAYER_USED=true                                        # true：注册阶段按 requirements.txt 构建 layer，跳过冷启动 pip 安装；false：远端冷启动安装

​

Prepare Data and Code

{
  "messages": [                          // 对话输入，ChatML 格式；每条 message 含 role 与 content
    {"role": "user", "content": "<您的业务问题>"}
  ],
  "rollout_extra": {"solution": "<参考答案或其他业务数据>"}  // 参考答案/业务数据，供 Reward 或自定义 Rollout 启用时消费；纯蒸馏仅透传不消费
}

​

Submit Training Task

import asyncio
from dashscope.finetune.agentic_rl import AgenticRL          # OPD 提交客户端
from dashscope.finetune.reinforcement import (
    TrainingDataset, ValidationDataset, DataSourceType,       # 训练/验证数据集及数据源类型
)

async def main():
    client = AgenticRL()
    result = await client.run(
        job_name="my-opd-job",                                # 任务名（控制台展示用）
        model="qwen3.5-9b",                       # 学生模型（基座模型，被蒸馏对象）
        teacher_model="your-teacher-model-id",     # 教师模型，OPD 唯一新增参数；传值即触发蒸馏
        training_datasets=[TrainingDataset(
            data_source_type=DataSourceType.FILE_ID,          # 以文件方式上传训练数据
            file_name="./data/train.jsonl")],
        validation_datasets=[ValidationDataset(
            data_source_type=DataSourceType.FILE_ID,          # 验证集同样以文件方式上传
            file_name="./data/validation.jsonl")],
        functions=None,                            # 纯蒸馏：不注册任何函数；Agentic OPD 在此传 Rollout/Reward 函数
        hyper_parameters={
            "n_epochs": 1, "n_rollouts": 8, "learning_rate": 1e-6,  # 蒸馏是微调模型已有能力，学习率起点 1e-6
            "eval_steps": 1, "ppo_mini_batch_size": 8,
            "lr_scheduler_type": "cosine",  # kl_loss_coef OPD 未启用，见训练配置
            # opd_teacher_* 部署参数见训练配置
        },
        resources={
            "charge_type": "mtu_postpaid",           # MTU 后付费计费
            "mtu_spec_code": "MTU4",                  # 训练单元规格
            "mtu_capacity": 24,                      # 容量（并发资源数）
        },
    )
    # HTTP 200 返回 job_id（查询训练状态的唯一标识）；否则打印原始响应便于排查
    print(result.output.job_id if result.status_code == 200 else result)

asyncio.run(main())                                  # 启动异步主函数

​

Getting Started Tips

• First choose an example: for plain text output tasks start from the Model OPD example; when output depends on API-level tool call Or multi-turn/custom interaction, start from the Agentic OPD example.
• Use streamlined data first: dozens of items + n_epochs=1 to verify the pipeline before switching to the full dataset, avoiding consuming MTU on the full dataset only to discover config errors later.
• Keep hyperparameters Default：learning_rate as starting point 1e-6(distillation fine-tunes the model's existing capabilities rather than rebuilding, too large a learning rate easily breaks output Format),lr_scheduler_type use cosine; when tuning, only change one variable at a time, at least train eval_steps × 3 steps before judging the trend.
• FAILED check logs first: Task Failed → log tab to view the error at the end, Or SDK Calling AgenticRL.logs(job_id="ft-xxx", lines=100)。

​

Post-training Process

​

Observation and Evaluation

• trace/distillation/valid_token_rate close to 1.0 (teacher model signal complete).
• actor/distillation/loss decreases Or stabilizes (student model converges toward teacher model).
• When Enable Reward, additionally watch validation/data/reward/mean@1 and trace/reward_metrics rise relative to baseline across all dimensions.

​

Publish and Deploy

​

Applicable Scenario

• Pure distillation (Basic): only teacher model + data, no function components written → Model OPD Development
• +Reward (advanced): add a scoring function, layering task reward on top of teacher model's per-token guidance (score by standard answer, score by multi-dimensional metrics) → Model OPD Development
• +Custom Rollout (advanced, Agentic OPD): when output depends on tool Calling/multi-turn interaction, customize the generation process; can be layered with Reward, or used alone → Agentic OPD Development

​

Next Steps

• Model OPD Development: complete details of function development (Rollout/Reward functions, Tracing integration).
• Agentic OPD Development: Custom Rollout (passing in the tools parameter, structured output), Agentic Reward development.
• Training Configuration: Submission Method, field-by-field descriptions, hyperparameters and Environment Variable.
• Observability Configuration: metric meanings, Trajectory viewing, acceptance criteria and troubleshooting decision tree.

​

FAQ

​

When to choose OPD over SFT

​

Can training effectiveness be judged without reward