Model OPD Development

Model OPD development only requires preparing data and submitting configuration to start training (pure distillation, no need to write function components) — student model output generation is completed by Bailian, no need to write. When online optimization by business goals is required, write a Reward scoring function to add to the training process (advanced usage). The differences between the two usages are as follows. For training overview see Online Policy Distillation Training Overview.

Dimension	No Reward (Pure Distillation)	With Reward
Development Work	Only prepare data and submit configuration, no need to write function components	Additionally write a Reward scoring function
Training Signal	Teacher model per-token probability distribution (sole supervision)	Teacher distribution + task reward (by standard answer or multi-dimensional scoring)
Applicable Scenario	The teacher model is already the task upper limit and does not need to be optimized online according to the task target	Need to be optimized online according to business goals, pulling toward the standard answer

Data format

One JSONL per line,messages is the conversation list,rollout_extra carries additional information.

{"messages":[{"role":"system","content":"<任务说明>"},{"role":"user","content":"<用户输入>"}],"rollout_extra":{"solution":"<ground_truth（标准答案）>"}}

In messages, system holds the task description, and user is the User Input. rollout_extra.solution stores the ground truth (standard answer) for Reward to read. Pure distillation does not register a Reward; rollout_extra serves only as a data marker and is not consumed. When Reward is enabled, the Reward reads the scoring basis from solution.

No Reward Development Reference (Pure Distillation)

Pure distillation does not write any functions; the development work is only to prepare data and submit configuration.

Submit Code

functions=None makes the control plane (scheduling layer) execute the native Rollout (Bailian's built-in sampling generation, where the student model directly generates output without passing the tools parameter; as opposed to Agentic OPD's multi-turn/custom interaction and custom Rollout). Teacher model distillation is the sole supervision signal. Pure distillation must pass None rather than an empty list [] (the control plane does not guarantee identical handling of the two).

from dashscope.finetune.agentic_rl import AgenticRL
from dashscope.finetune.reinforcement import (
    DataSourceType, TrainingDataset, ValidationDataset,
)

client = AgenticRL()
result = await client.run(
    job_name="my-pure-distill",  # 任务名，唯一标识本次训练
    model=student_model,  # 学生模型 ID，被蒸馏的对象
    teacher_model=teacher_model,  # 教师模型 ID，提供 token 概率分布作为蒸馏监督信号
    training_datasets=[
        TrainingDataset(
            data_source_type=DataSourceType.FILE_ID,  # 数据来源类型：文件 ID（百炼已上传的文件）
            file_name="data/train.jsonl",  # 训练集路径，JSONL 格式
        )
    ],
    validation_datasets=[
        ValidationDataset(
            data_source_type=DataSourceType.FILE_ID,
            file_name="data/validation.jsonl",  # 验证集路径，用于训练中评估
        )
    ],
    functions=None,  # None 而非空列表：控制面据此执行原生 Rollout（纯蒸馏，不注册任何函数组件）
    hyper_parameters={...},  # 超参、opd_teacher_* 部署参数见训练配置
)

Stacking Reward (Advanced Usage)

After registering the Reward Component, the control plane invokes it to score after each Rollout, Outputting per-dimension metrics to participate in training.

Reward Function Structure

Inherit AbstractRewardProcessor, implement async process(input: RewardInput) -> RewardOutput, key conventions are as follows:

Decorator automatically connects to Tracing: process uses the @observe_processor decorator; the trace client is implicitly injected by the decorator and is not explicitly called within the process method
Input Field: RewardInput contains ground_truth and agent_output (native Rollout output, read-only)
Output Fields: RewardOutput contains Reward(reward_score, reward_metrics) and TaskStatus

from dashscope.finetune.reinforcement import (
    AbstractRewardProcessor, Reward, RewardInput, RewardOutput, TaskStatus,
)
from dashscope.finetune.reinforcement.component.observability import observe_processor

class MyRewardProcessor(AbstractRewardProcessor):
    @observe_processor  # 装饰器：自动接入 Tracing，追踪客户端隐式注入，方法内不显式调用
    async def process(self, input: RewardInput) -> RewardOutput:
        # 入参 input: RewardInput，含 ground_truth（评分基准）与 agent_output（原生 Rollout 产出，只读）
        # 返回 RewardOutput，含 Reward(reward_score, reward_metrics) 与 TaskStatus
        # 1. 取数据：ground_truth + 末轮 assistant 文本
        ground_truth = input.ground_truth  # 优先取注入的标准答案
        if ground_truth is None:
            if input.agent_output.rollout_extra:  # 回退到数据侧 rollout_extra.solution
                ground_truth = input.agent_output.rollout_extra.get("solution")
        messages = input.agent_output.messages  # Rollout 对话历史
        content = messages[-1]["content"] if messages else ""  # 末轮 assistant 回答文本
        # 2. 剥离思考链 thinking（见输出文本清理）
        content = strip_reasoning(content)
        # 3. 评分：返回 dict，含主分 reward_score 与附加维度
        metrics = your_scoring_function(content, ground_truth)
        # 4. reward_score 单独赋给 Reward.reward_score，其余进 reward_metrics
        reward_score = metrics.pop("reward_score")  # 从 dict 取出主分，避免重复进 metrics
        return RewardOutput(
            reward=Reward(reward_score=reward_score, reward_metrics=metrics),  # 主分 + 维度 dict[float]
            status=TaskStatus.SUCCESS, error=None,  # 处理结果状态，失败置 TaskStatus.FAIL 并填 error
        )

RewardInput / RewardOutput key fields:

Field	Source	Usage
`ground_truth`	RewardInput (sourced from rollout_extra.solution)	Scoring Basis
`agent_output.messages`	Native Rollout Output	Take the last round of assistant content
`reward_score`	Reward.reward_score	Main score, reported separately
`reward_metrics`	Reward.reward_metrics	Other dimensions dict[float]
`status`	TaskStatus	Processing result status

Output Text Cleanup

In the native Rollout output, besides the student's answer, the thinking process and tool call format text are also mixed in. If not cleaned, they will interfere with Reward scoring; you need to first strip these out and leave only the answer before scoring. Cleanup steps:

Strip the thinking chain thinking
Remove tool call Text

import re
# 占位标签示意，实际正则按业务定义
# 实际标签名取自学生模型的 chat template（如 <think>...</think>），按实际模型定义正则
THINKING_BLOCK = re.compile(r"imd.*?imd_end", re.DOTALL)  # 匹配完整 thinking 块（含闭标签）
LEADING_THINK = re.compile(r"^.*?imd_end", re.DOTALL)  # 首个闭标签前可能有散落 thinking，先删完整块再取 tail
TOOL_CALL_BLOCK = re.compile(r"tool_call.*?end_tool_call|function=.*?end_function", re.DOTALL)  # 工具调用文本块

def strip_reasoning(content):
    # 入参 content: 末轮 assistant 原始文本；返回: 仅保留回答正文
    # 剥完整 thinking 块 + 首个闭标签前内容 + 工具调用文本
    content = THINKING_BLOCK.sub("", content)  # 先删成对 thinking 块
    content = LEADING_THINK.sub("", content)  # 再删首个闭标签前的散落 thinking
    content = TOOL_CALL_BLOCK.sub("", content)  # 最后删工具调用文本
    return content

Scoring Logic

reward_score as the main reward scalar,reward_metrics as the additional dimension dictionary (each value float).reward_score assigned separately to Reward.reward_score, not duplicated into reward_metrics (metrics.pop("reward_score")). Dimension division and weights are determined by the task, implement according to business needs. Below is the minimal runnable implementation: content containing ground_truth gets 1.0, otherwise 0.0, convenient for first running through the pipeline then expanding dimensions.

def your_scoring_function(content, ground_truth):
    # 入参 content: 清理后的回答文本；ground_truth: 标准答案（评分基准）
    # 返回 dict：reward_score 为主分（必填，被 pop 单独上报），其余键为附加维度
    # 最小实现：content 含 ground_truth 即得 1.0，否则 0.0
    hit = float(ground_truth in content) if ground_truth else 0.0  # 命中标记，无 ground_truth 则 0
    return {
        "reward_score": hit,  # 主奖励标量，单独赋给 Reward.reward_score
        "coverage": hit,  # 附加维度示例：覆盖率，扩展维度后自行调整
    }

Submit the training task with Reward enabled

After writing MyRewardProcessor after, use RewardFunctionComponent register it to functions list.run() One step to complete registration → Upload Data → submit task.

from dashscope.finetune.agentic_rl import AgenticRL
from dashscope.finetune.reinforcement import (
    RewardFunctionComponent, FunctionComponentModel,
    FunctionComponentRuntime, TrainingDataset,
    ValidationDataset, DataSourceType,
)

client = AgenticRL()
result = await client.run(
    job_name="my-opd-reward-job",  # 任务名
    model=student_model,  # 学生模型 ID
    teacher_model=teacher_model,  # 教师模型 ID，蒸馏监督信号
    training_datasets=[TrainingDataset(
        data_source_type=DataSourceType.FILE_ID,
        file_name="data/train.jsonl")],  # 训练集
    validation_datasets=[ValidationDataset(
        data_source_type=DataSourceType.FILE_ID,
        file_name="data/validation.jsonl")],  # 验证集
    functions=[RewardFunctionComponent(  # 注册 Reward 组件，控制面在每条 Rollout 后调用评分
        name="reward-1",  # 组件名，唯一标识
        weight=1.0,  # 奖励权重，多 Reward 时按加权聚合
        timeout=120,  # 单次调用超时（秒），超时按失败处理
        fcmodel=FunctionComponentModel(
            zipdir=".",  # 工作区根目录（含 functions/）
            classpath="functions.reward.reward.MyRewardProcessor"),  # 入口类（示例类名，实际由您自定义）
        runtime=FunctionComponentRuntime(  # 组件运行时资源配置
            cpu=2, memory_size=4096, disk_size=512,  # CPU/内存(MB)/磁盘(MB)
            concurrency=20, capacity=10, min_capacity=10, max_capacity=30))],  # 并发与弹性容量配置
    hyper_parameters={...},  # 超参、opd_teacher_* 部署参数见训练配置
)

classpath points to the module path where MyRewardProcessor is located; zipdir="." packages and uploads the entire workspace. For complete field descriptions, see Training Configuration.

Function Testing

Before submitting for training, first locally verify the Reward scoring logic to avoid discovering scoring errors only within the training task (wasting compute power). For your_scoring_function, input sample content and ground_truth, asserting that each dimension's value meets expectations. Place the following test and your_scoring_function and strip_reasoning into the same file (e.g. test_reward.py), and it can run locally.

def test_hit():
    # 用例：content 含 ground_truth → 命中，主分与覆盖率均为 1.0
    metrics = your_scoring_function("query_status done", "query_status")  # 入参: 回答文本、标准答案
    assert metrics["reward_score"] == 1.0  # 主分单独校验
    assert metrics["coverage"] == 1.0  # 附加维度校验

def test_thinking_residual():
    # 用例：带思考链残留，strip 后含 ground_truth → 命中
    content = "imd thinking imd_end query_status"  # 含 thinking 块的原始输出
    cleaned = strip_reasoning(content)  # 清理后应只剩 "query_status"
    metrics = your_scoring_function(cleaned, "query_status")
    assert metrics["coverage"] == 1.0  # 验证清理 + 评分链路打通

def test_miss():
    # 用例：content 不含 ground_truth → 主分 0.0
    metrics = your_scoring_function("delete_item failed", "query_status")
    assert metrics["reward_score"] == 0.0

Run the test:

python -m pytest test_reward.py -v

The test covers scenarios such as hit, residual chain of thought, and miss. After extending the dimensions, add corresponding assertions.

Next Steps

Training Configuration (hyperparameters, Resources, Data Source) see Online Policy Distillation Training Configuration
Observability (Tracing, metrics dashboard) see Online Policy Distillation Observability Configuration

FAQ

Show reward_metrics not displayed

Check reward_score whether it is taken out via pop and assigned separately to Reward.reward_score, and whether the rest of the dimension values are all float.

Show Score is 0 (coverage not hit)

The native Rollout parser does not execute, content retains thinking and tool call text; you need to first strip_reasoning clean it, then according to the dimensions you defined (such as coverage) to judge, rather than directly judging the original content.

Show teacher_model error

Common causes: teacher_model filled in a model ID not registered in Bailian (such as a HuggingFace repository name). Troubleshooting: switch to a model ID supported by Bailian, for the complete list and more reasons see Training Configuration FAQ.

Show ground_truth not available (empty)

Check each JSONL entry in the data rollout_extra.solution whether it has a value. Reward is first taken from input.ground_truth , falling back to input.agent_output.rollout_extra.solution when missing; if both are empty, the score is 0.

Show Local test reports ModuleNotFoundError

Confirm functions/ directory contains __init__.py(making the directory an importable Python package); classpath aligns with the file path, e.g. functions.reward.my_reward.MyRewardProcessor corresponds to functions/reward/my_reward.py inside MyRewardProcessor Class.

​Data format

​No Reward Development Reference (Pure Distillation)

​Submit Code

​Stacking Reward (Advanced Usage)

​Reward Function Structure

​Output Text Cleanup

​Scoring Logic

​Submit the training task with Reward enabled

​Function Testing

​Next Steps

​FAQ