Model OPD Development

> ## Documentation Index > Fetch the complete documentation index at: https://docs.bailian.console.aliyun.com/llms.txt > Use this file to discover all available pages before exploring further. # Model OPD Development > Model OPD Development: pure distillation requires no function components; stacking Reward (advanced usage) involves scoring function development, with data formats and FAQs Model OPD development only requires preparing data and submitting configuration to start training (pure distillation, no need to write function components) — student model output generation is completed by Bailian, no need to write. When online optimization by business goals is required, write a Reward scoring function to add to the training process (advanced usage). The differences between the two usages are as follows. For training overview see [Online Policy Distillation Training Overview](opd-training-overview.md).

Dimension	No Reward (Pure Distillation)	With Reward
Development Work	Only prepare data and submit configuration, no need to write function components	Additionally write a Reward scoring function
Training Signal	Teacher model per-token probability distribution (sole supervision)	Teacher distribution + task reward (by standard answer or multi-dimensional scoring)
Applicable Scenario	The teacher model is already the task upper limit and does not need to be optimized online according to the task target	Need to be optimized online according to business goals, pulling toward the standard answer

## Data format One JSONL per line,`messages` is the conversation list,`rollout_extra` carries additional information. ```plaintext {"messages":[{"role":"system","content":"<任务说明>"},{"role":"user","content":"<用户输入>"}],"rollout_extra":{"solution":""}} ``` In `messages`, `system` holds the task description, and `user` is the User Input. `rollout_extra.solution` stores the ground truth (standard answer) for Reward to read. Pure distillation does not register a Reward; `rollout_extra` serves only as a data marker and is not consumed. When Reward is enabled, the Reward reads the scoring basis from `solution`. ## No Reward Development Reference (Pure Distillation) Pure distillation does not write any functions; the development work is only to prepare data and submit configuration. ### Submit Code `functions=None` makes the control plane (scheduling layer) execute the native Rollout (Bailian's built-in sampling generation, where the student model directly generates output without passing the tools parameter; as opposed to Agentic OPD's multi-turn/custom interaction and custom Rollout). Teacher model distillation is the sole supervision signal. Pure distillation must pass `None` rather than an empty list `[]` (the control plane does not guarantee identical handling of the two). ```plaintext from dashscope.finetune.agentic_rl import AgenticRL from dashscope.finetune.reinforcement import ( DataSourceType, TrainingDataset, ValidationDataset, ) client = AgenticRL() result = await client.run( job_name="my-pure-distill", # 任务名，唯一标识本次训练 model=student_model, # 学生模型 ID，被蒸馏的对象 teacher_model=teacher_model, # 教师模型 ID，提供 token 概率分布作为蒸馏监督信号 training_datasets=[ TrainingDataset( data_source_type=DataSourceType.FILE_ID, # 数据来源类型：文件 ID（百炼已上传的文件） file_name="data/train.jsonl", # 训练集路径，JSONL 格式 ) ], validation_datasets=[ ValidationDataset( data_source_type=DataSourceType.FILE_ID, file_name="data/validation.jsonl", # 验证集路径，用于训练中评估 ) ], functions=None, # None 而非空列表：控制面据此执行原生 Rollout（纯蒸馏，不注册任何函数组件） hyper_parameters={...}, # 超参、opd_teacher_* 部署参数见训练配置 ) ``` ## Stacking Reward (Advanced Usage) After registering the Reward Component, the control plane invokes it to score after each Rollout, Outputting per-dimension metrics to participate in training. ### Reward Function Structure Inherit `AbstractRewardProcessor`, implement `async process(input: RewardInput) -> RewardOutput`, key conventions are as follows: - **Decorator automatically connects to Tracing**: `process` uses the `@observe_processor` decorator; the trace client is implicitly injected by the decorator and is not explicitly called within the process method - **Input Field**: `RewardInput` contains `ground_truth` and `agent_output` (native Rollout output, read-only) - **Output Fields**: `RewardOutput` contains `Reward(reward_score, reward_metrics)` and `TaskStatus` ```plaintext from dashscope.finetune.reinforcement import ( AbstractRewardProcessor, Reward, RewardInput, RewardOutput, TaskStatus, ) from dashscope.finetune.reinforcement.component.observability import observe_processor class MyRewardProcessor(AbstractRewardProcessor): @observe_processor # 装饰器：自动接入 Tracing，追踪客户端隐式注入，方法内不显式调用 async def process(self, input: RewardInput) -> RewardOutput: # 入参 input: RewardInput，含 ground_truth（评分基准）与 agent_output（原生 Rollout 产出，只读） # 返回 RewardOutput，含 Reward(reward_score, reward_metrics) 与 TaskStatus # 1. 取数据：ground_truth + 末轮 assistant 文本 ground_truth = input.ground_truth # 优先取注入的标准答案 if ground_truth is None: if input.agent_output.rollout_extra: # 回退到数据侧 rollout_extra.solution ground_truth = input.agent_output.rollout_extra.get("solution") messages = input.agent_output.messages # Rollout 对话历史 content = messages[-1]["content"] if messages else "" # 末轮 assistant 回答文本 # 2. 剥离思考链 thinking（见输出文本清理） content = strip_reasoning(content) # 3. 评分：返回 dict，含主分 reward_score 与附加维度 metrics = your_scoring_function(content, ground_truth) # 4. reward_score 单独赋给 Reward.reward_score，其余进 reward_metrics reward_score = metrics.pop("reward_score") # 从 dict 取出主分，避免重复进 metrics return RewardOutput( reward=Reward(reward_score=reward_score, reward_metrics=metrics), # 主分 + 维度 dict[float] status=TaskStatus.SUCCESS, error=None, # 处理结果状态，失败置 TaskStatus.FAIL 并填 error ) ``` `RewardInput` / `RewardOutput` key fields:

Field	Source	Usage
`ground\_truth`	RewardInput (sourced from rollout\_extra.solution)	Scoring Basis
`agent\_output.messages`	Native Rollout Output	Take the last round of assistant content
`reward\_score`	Reward.reward\_score	Main score, reported separately
`reward\_metrics`	Reward.reward\_metrics	Other dimensions dict\[float]
`status`	TaskStatus	Processing result status

### Output Text Cleanup In the native Rollout output, besides the student's answer, the thinking process and tool call format text are also mixed in. If not cleaned, they will interfere with Reward scoring; you need to first strip these out and leave only the answer before scoring. Cleanup steps: 1. Strip the thinking chain thinking 2. Remove tool call Text ```plaintext import re # 占位标签示意，实际正则按业务定义 # 实际标签名取自学生模型的 chat template（如 ...），按实际模型定义正则 THINKING_BLOCK = re.compile(r"imd.*?imd_end", re.DOTALL) # 匹配完整 thinking 块（含闭标签） LEADING_THINK = re.compile(r"^.*?imd_end", re.DOTALL) # 首个闭标签前可能有散落 thinking，先删完整块再取 tail TOOL_CALL_BLOCK = re.compile(r"tool_call.*?end_tool_call|function=.*?end_function", re.DOTALL) # 工具调用文本块 def strip_reasoning(content): # 入参 content: 末轮 assistant 原始文本；返回: 仅保留回答正文 # 剥完整 thinking 块 + 首个闭标签前内容 + 工具调用文本 content = THINKING_BLOCK.sub("", content) # 先删成对 thinking 块 content = LEADING_THINK.sub("", content) # 再删首个闭标签前的散落 thinking content = TOOL_CALL_BLOCK.sub("", content) # 最后删工具调用文本 return content ``` ### Scoring Logic `reward_score` as the main reward scalar,`reward_metrics` as the additional dimension dictionary (each value float).`reward_score` assigned separately to `Reward.reward_score`, not duplicated into `reward_metrics` (`metrics.pop("reward_score")`). Dimension division and weights are determined by the task, implement according to business needs. Below is the minimal runnable implementation: content containing ground\_truth gets 1.0, otherwise 0.0, convenient for first running through the pipeline then expanding dimensions. ```plaintext def your_scoring_function(content, ground_truth): # 入参 content: 清理后的回答文本；ground_truth: 标准答案（评分基准） # 返回 dict：reward_score 为主分（必填，被 pop 单独上报），其余键为附加维度 # 最小实现：content 含 ground_truth 即得 1.0，否则 0.0 hit = float(ground_truth in content) if ground_truth else 0.0 # 命中标记，无 ground_truth 则 0 return { "reward_score": hit, # 主奖励标量，单独赋给 Reward.reward_score "coverage": hit, # 附加维度示例：覆盖率，扩展维度后自行调整 } ``` ### Submit the training task with Reward enabled After writing `MyRewardProcessor` after, use `RewardFunctionComponent` register it to `functions` list.`run()` One step to complete registration → Upload Data → submit task. ```plaintext from dashscope.finetune.agentic_rl import AgenticRL from dashscope.finetune.reinforcement import ( RewardFunctionComponent, FunctionComponentModel, FunctionComponentRuntime, TrainingDataset, ValidationDataset, DataSourceType, ) client = AgenticRL() result = await client.run( job_name="my-opd-reward-job", # 任务名 model=student_model, # 学生模型 ID teacher_model=teacher_model, # 教师模型 ID，蒸馏监督信号 training_datasets=[TrainingDataset( data_source_type=DataSourceType.FILE_ID, file_name="data/train.jsonl")], # 训练集 validation_datasets=[ValidationDataset( data_source_type=DataSourceType.FILE_ID, file_name="data/validation.jsonl")], # 验证集 functions=[RewardFunctionComponent( # 注册 Reward 组件，控制面在每条 Rollout 后调用评分 name="reward-1", # 组件名，唯一标识 weight=1.0, # 奖励权重，多 Reward 时按加权聚合 timeout=120, # 单次调用超时（秒），超时按失败处理 fcmodel=FunctionComponentModel( zipdir=".", # 工作区根目录（含 functions/） classpath="functions.reward.reward.MyRewardProcessor"), # 入口类（示例类名，实际由您自定义） runtime=FunctionComponentRuntime( # 组件运行时资源配置 cpu=2, memory_size=4096, disk_size=512, # CPU/内存(MB)/磁盘(MB) concurrency=20, capacity=10, min_capacity=10, max_capacity=30))], # 并发与弹性容量配置 hyper_parameters={...}, # 超参、opd_teacher_* 部署参数见训练配置 ) ``` `classpath` points to the module path where `MyRewardProcessor` is located; `zipdir="."` packages and uploads the entire workspace. For complete field descriptions, see [Training Configuration](opd-training-config.md). ### Function Testing Before submitting for training, first locally verify the Reward scoring logic to avoid discovering scoring errors only within the training task (wasting compute power). For `your_scoring_function`, input sample content and ground\_truth, asserting that each dimension's value meets expectations. Place the following test and `your_scoring_function` and `strip_reasoning` into the same file (e.g. `test_reward.py`), and it can run locally. ```plaintext def test_hit(): # 用例：content 含 ground_truth → 命中，主分与覆盖率均为 1.0 metrics = your_scoring_function("query_status done", "query_status") # 入参: 回答文本、标准答案 assert metrics["reward_score"] == 1.0 # 主分单独校验 assert metrics["coverage"] == 1.0 # 附加维度校验 def test_thinking_residual(): # 用例：带思考链残留，strip 后含 ground_truth → 命中 content = "imd thinking imd_end query_status" # 含 thinking 块的原始输出 cleaned = strip_reasoning(content) # 清理后应只剩 "query_status" metrics = your_scoring_function(cleaned, "query_status") assert metrics["coverage"] == 1.0 # 验证清理 + 评分链路打通 def test_miss(): # 用例：content 不含 ground_truth → 主分 0.0 metrics = your_scoring_function("delete_item failed", "query_status") assert metrics["reward_score"] == 0.0 ``` Run the test: ```plaintext python -m pytest test_reward.py -v ``` The test covers scenarios such as hit, residual chain of thought, and miss. After extending the dimensions, add corresponding assertions. ## Next Steps - Training Configuration (hyperparameters, Resources, Data Source) see [Online Policy Distillation Training Configuration](opd-training-config.md) - Observability (Tracing, metrics dashboard) see [Online Policy Distillation Observability Configuration](opd-observable-config.md) ## FAQ Check `reward_score` whether it is taken out via pop and assigned separately to `Reward.reward_score`, and whether the rest of the dimension values are all float. The native Rollout parser does not execute, content retains thinking and tool call text; you need to first `strip_reasoning` clean it, then according to the dimensions you defined (such as `coverage`) to judge, rather than directly judging the original content. Common causes: `teacher_model` filled in a model ID not registered in Bailian (such as a HuggingFace repository name). Troubleshooting: switch to a model ID supported by Bailian, for the complete list and more reasons see [Training Configuration FAQ](opd-training-config.md#opd-faq). Check each JSONL entry in the data `rollout_extra.solution` whether it has a value. Reward is first taken from `input.ground_truth` , falling back to `input.agent_output.rollout_extra.solution` when missing; if both are empty, the score is 0. Confirm `functions/` directory contains `__init__.py`(making the directory an importable Python package); `classpath` aligns with the file path, e.g. `functions.reward.my_reward.MyRewardProcessor` corresponds to `functions/reward/my_reward.py` inside `MyRewardProcessor` Class.