Online Policy Distillation Observable Configuration and Metrics Reference

• OPD principles and example selection → see OPD Training Overview
• Model OPD development → see Model OPD Development; Agentic OPD development → see Agentic OPD Development
• Full set of submission parameters and hyperparameters → see OPD Training Configuration

​

Overview

1. Integrate Tracing — the SDK automatically reports training metrics to the console; how to view them on the Trajectory page
2. Custom metrics — reporting paths and constraints for rollout_metrics and reward_metrics
3. View effects in the console — how to read the Metrics / Trajectory / Output / Logs tabs
4. Training metrics reference — OPD-specific metric dictionary + common metrics quick reference
5. Judge training effects and troubleshoot — in-training interpretation, post-training evaluation, and FAQ

​

Integrate Tracing

​

Step 1: Add dependencies

​

Step 2: Add observability code

# 调用链结构：装饰器自动嵌套形成 Span 树，[ENTRY] 为入口 Span，[LLM]/[TOOL]/[custom] 为子 Span
[ENTRY: ROLLOUT] @observe_processor          ← Rollout 处理器入口
├── [LLM] trace_client / @observe_llm        ← LLM 调用（学生模型调用）
│   └── (OpenAI / LangChain / DashScope API)
├── [TOOL] trace_tool / @observe_tool        ← 工具调用
│   ├── tool: my_tool (MCP)
│   └── tool: my_scorer (自定义)
└── [custom] rollout_metrics                 ← Rollout 自定义指标

[ENTRY: REWARD] @observe_processor           ← Reward 处理器入口
├── [LLM] trace_client / @observe_llm        ← LLM 调用（评分用）
├── [TOOL] trace_tool / @observe_tool        ← 工具调用
└── [custom] reward_metrics                  ← Reward 自定义指标

​

@observe_processor — Track processor entry

from dashscope.finetune.reinforcement.component.observability import observe_processor
# 导入 Rollout 处理器基类与输入输出类型（项目内已定义）

class MyRolloutProcessor(AbstractRolloutProcessor):
    @observe_processor  # Span 类型 = ROLLOUT（按继承 AbstractRolloutProcessor 自动推断）
    async def process(self, input: RolloutInput) -> RolloutOutput:
        # input: Rollout 输入，含 prompt、模型资源、工具列表等
        await self._async_setup()  # 异步初始化：建 LLM 客户端、加载工具
        return await self._async_process(input)  # 执行实际 Rollout 逻辑，返回轨迹与指标

class MyRewardProcessor(AbstractRewardProcessor):
    @observe_processor  # Span 类型 = REWARD（按继承 AbstractRewardProcessor 自动推断）
    async def process(self, input: RewardInput) -> RewardOutput:
        # input: Reward 输入，含 agent_output（模型轨迹）与 ground_truth（标准答案）
        messages = input.agent_output.messages  # 取模型多轮对话消息列表
        content = messages[-1].get("content", "") if messages else ""  # 取最后一条 assistant 回复文本
        score = await evaluate(content, input.ground_truth)  # 用标准答案对回复评分，返回主分
        return RewardOutput(
            reward=Reward(reward_score=score, reward_metrics={...}),  # 主分入 reward_score，分维度入 reward_metrics
            status=TaskStatus.SUCCESS,  # 任务状态：SUCCESS 表示评分成功
            error=None,  # 无异常时为 None；失败时填错误信息
        )

​

trace_client() — Track LLM client

• OpenAI clients (AsyncOpenAI / OpenAI)
• OpenAI completions resource (.chat.completions)
• LangChain ChatOpenAI and similar classes (via .client / .async_client)
• DashScope Generation class (pass the class itself, not an instance)

from dashscope.finetune.reinforcement.component.observability import trace_client
# 导入 LLM 客户端包装函数，调用后该客户端的请求自动产生 LLM Span

class MyRolloutProcessor(AbstractRolloutProcessor):
    def _build_llm(self, input: RolloutInput) -> ChatOpenAI:
        # input: Rollout 输入，model_resource 含模型名等资源配置
        resource = input.model_resource  # 取模型资源（由 Runtime 注入的学生模型）
        llm = ChatOpenAI(model=resource.model_name, ...)  # 构造 LangChain ChatOpenAI 客户端
        trace_client(llm)  # 包装后自动追踪所有 LLM 调用（记录模型名、请求、Token 用量、延迟）
        return llm

​

trace_tool() — Track tool calls

• A single LangChain BaseTool
• List / tuple / dict (auto-traversed)
• LangGraph ToolNode (auto-expands .tools_by_name)
• MCP tools (auto-detected, provider set to "mcp")
• Custom provider (trace_tool(tool, provider="my-plugin"))

​

@observe_llm — Custom LLM function

from dashscope.finetune.reinforcement.component.observability import observe_llm
# 导入 LLM 装饰器，用于手动标记无法被 trace_client 自动识别的 LLM 调用

class MyRolloutProcessor(AbstractRolloutProcessor):
    @observe_llm  # 标记为 LLM Span（记录模型名、messages、Token 用量）
    async def _call_llm(self, *, messages: List[Dict], model: str) -> Any:
        # messages: 对话消息列表；model: 模型名（* 后关键字参数为签名硬性要求）
        ...

​

@observe_tool — Custom tool function

from dashscope.finetune.reinforcement.component.observability import observe_tool
# 导入工具装饰器，用于手动标记无法被 trace_tool 自动识别的工具函数

class MyRolloutProcessor(AbstractRolloutProcessor):
    @observe_tool(name="my_scorer")  # 标记为 TOOL Span，name 自定义 Span 显示名
    def _score_response(self, *, messages: List[Dict]) -> float:
        # messages: 对话消息列表，函数返回评分为 float
        ...

​

Step 3: Submit the task

# FunctionComponentRuntime: Rollout/Reward 组件运行时配置
rollout_runtime = FunctionComponentRuntime(
    cpu=2, memory_size=4096, disk_size=512,  # 资源：2 核 CPU、4GB 内存、512MB 磁盘
    concurrency=30, capacity=30,  # 并发 30、容量 30（单实例同时处理请求数）
    min_capacity=30, max_capacity=60,  # 弹性伸缩下限 30、上限 60
    env={}  # 留空 = 默认开启 Tracing；设 {"ENABLE_TRAJECTORY": "false"} 可关闭
)

​

Tracing performance and cost trade-off

​

Data pipeline and cost sources

• Data flow: function code (decorators) → OpenTelemetry SDK → ARMS → console Trajectory/Metrics tabs
• Cost sources: ARMS Span storage (pay-as-you-go) + a small amount of CPU/network overhead on the function side + a slight increase in training latency

​

Switch strategy and cost governance

• Count and cardinality: the count and cardinality of reward_metrics / rollout_metrics affect ARMS storage
• Avoid high-cardinality fields: do not use user_id / request_id as metric keys
• Streamline metrics: keep key metrics ≤10; do not leave temporary debugging metrics

​

Custom metrics

• Flat dict: reward_metrics / rollout_metrics return a flat dict (not a nested list), with each value a float
• Main score reported separately: reward_score goes into Reward.reward_score alone; do not duplicate it in reward_metrics
• Non-float only persisted: non-float data is persisted only via rollout_extra and does not enter metrics
• Metric path: the reward_score main score maps to trace/reward/<name>/, and each dimension maps to trace/reward_metrics/<name>/<metric>

​

View effects in the console

​

Trajectory and behavior analysis

• Trace list: shows all sampled trajectories; supports filtering by Sample ID / Trajectory ID / Epoch / Step
• Dialogue process: complete multi-turn interaction (user → assistant → tool_call → tool_result → assistant), intuitively showing the model's reasoning chain
• Reward score: each Step shows the corresponding Reward score and status (SUCCESS/FAILED)

• Tool call records: tool name, call parameters, return results, and duration
• Tracing sub-tab: the Span tree of each trajectory; expandable to view the full details of each tool call and LLM request

• Sample: one raw sample in the training data (one problem, one instruction, one prompt)
• Trajectory: one specific interaction trajectory produced by the same Sample under n_rollouts samplings
• Relationship: one Sample → N Trajectories (N = n_rollouts)

• Step dimension: select a training Step to view the Reward aggregation (average score, success rate, trend chart) of all samples under that Step, to judge overall training trend
• Sample dimension: select a Sample ID to compare the Reward of the same sample across different trajectories, to find problematic samples
• Trajectory dimension: view the raw score of each scoring dimension for a single trajectory, for attribution analysis

​

Metrics, output, and logs

1. Select the target Checkpoint and click the Publish button
2. Wait for publishing to complete (status changes from "Pending Publish" to "Published")
3. After publishing, you can call it via the model name in the API

• SDK: AgenticRL.logs(job_id="ft-xxx", lines=100)
• CLI: dashscope rl logs "ft-xxx" --lines 100

​

Training metrics reference

​

OPD-specific metrics

​

Key metrics quick reference

​

Common metrics

​

Judge training effects

​

In-training interpretation order

​

Post-training business evaluation

​

Acceptance criteria

​

OPD troubleshooting decision tree

​

P1 Distillation loss not declining

• Main signal: actor/distillation/loss does not decline over training or even rises
• Root cause: teacher model signal missing / learning rate unsuitable (too small to converge or too large to break the output format) / poor data quality
• Handling: first check whether valid_token_rate is close to 1.0 → when the signal is normal, loss not declining is often due to a too-small learning rate; increase learning_rate (×1.5~2, see the training config tuning decision table) → spot-check the reference answers in rollout_extra of the training data

​

P2 Teacher model scoring missing

• Main signal: trace/distillation/valid_token_rate is noticeably below 1.0
• Root cause: scoring missing on the teacher model side (the teacher model is hosted on the server, opaque to the user, and cannot be troubleshot directly)
• Handling: check whether missing_token_rate and empty_response_rate rise in tandem → if the teacher model timeout or empty-response ratio is high, submit a ticket

​

P3 Validation set not rising

• Main signal: validation/data/reward/mean@1 shows a downward inflection while training reward keeps rising
• General handling: early stop (select an earlier Checkpoint on the Output tab) + keep pre- and post-training evaluation criteria consistent + view reward_metrics regression per dimension
• Additional root cause: overfitting the teacher model style (in pure distillation)
• Additional handling: spot-check the reference answer quality in rollout_extra

​

Next steps

• OPD Training Overview → principles and end-to-end flow
• Model OPD Development / Agentic OPD Development → full function development details
• OPD Training Configuration → full set of submission parameters and hyperparameters

​

FAQ

​

Teacher model side anomalies

​

Self-check items

​

Fault quick reference and FAILED troubleshooting

1. Step 1 Task list confirm status and the failure time point
2. Step 2 Logs tab last 100-500 lines (SDK AgenticRL.logs(job_id, lines=100) / CLI dashscope rl logs --lines 100)
3. Step 3 Distinguish error layer:
   • User function error (Rollout/Reward throws an exception) → test_functions to reproduce locally → fix code → re-register/run
   • Framework error (OOM / insufficient resources / network) → tune concurrency / capacity
   • Data error (JSONL parse failed) → validate single-line format and rollout_extra