> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bailian.console.aliyun.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Online Policy Distillation Observable Configuration and Metrics Reference

> Observability handbook for OPD training: Tracing integration, custom metric reporting, console tabs, metric dictionary, training effect evaluation, and FAQ troubleshooting

<meta property="og:title" content="Online Policy Distillation Observable Configuration and Metrics Reference" />

Online Policy Distillation (OPD) training automatically records every LLM call, tool call, and scoring detail, and visualizes them in the Model Studio console. Integration only requires adding a few decorators and wrapper calls in your function code.

<strong>Related documentation:</strong>

- OPD principles and example selection → see [OPD Training Overview](opd-training-overview.md)
- Model OPD development → see [Model OPD Development](opd-model-development-guide.md); Agentic OPD development → see [Agentic OPD Development](opd-agentic-development-guide.md)
- Full set of submission parameters and hyperparameters → see [OPD Training Configuration](opd-training-config.md)

## Overview <span id="opd-obs-overview" />

The observability capability is built on <strong>OpenTelemetry</strong> (an open-source observability standard) to implement Tracing (distributed tracing). Data is exported to <strong>ARMS (Application Real-time Monitoring Service)</strong> and displayed in the Model Studio console. A <strong>Span</strong> is the observability unit of a single operation in OpenTelemetry; multiple Spans nest automatically to form a call chain.

Five sections:

1. Integrate Tracing — the SDK automatically reports training metrics to the console; how to view them on the Trajectory page
2. Custom metrics — reporting paths and constraints for `rollout_metrics` and `reward_metrics`
3. View effects in the console — how to read the Metrics / Trajectory / Output / Logs tabs
4. Training metrics reference — OPD-specific metric dictionary + common metrics quick reference
5. Judge training effects and troubleshoot — in-training interpretation, post-training evaluation, and FAQ

## Integrate Tracing <span id="opd-tracing" />

Using the Rollout processor of an OPD project as an example, Tracing is integrated in 3 steps. <strong>OPD difference:</strong> the LLM Span in the call chain corresponds to the student model call, and teacher model scoring happens on the server side.

### Step 1: Add dependencies <span id="opd-tracing-step1" />

Add the following OpenTelemetry-related dependencies to the `requirements.txt` in the project root:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "50%" }} /><col style={{ width: "50%" }} /></colgroup><thead><tr><th><p><strong>Dependency</strong></p></th><th><p><strong>Description</strong></p></th></tr></thead><tbody><tr><td><p><code>opentelemetry-api==1.41.1</code></p></td><td><p>OTel API</p></td></tr><tr><td><p><code>opentelemetry-sdk==1.41.1</code></p></td><td><p>OTel SDK</p></td></tr><tr><td><p><code>opentelemetry-exporter-otlp-proto-http==1.41.1</code></p></td><td><p>OTLP HTTP exporter</p></td></tr><tr><td><p><code>opentelemetry-processor-baggage==0.62b1</code></p></td><td><p>Baggage processor</p></td></tr><tr><td><p><code>loongsuite-util-genai==0.4.0</code></p></td><td><p>Generative AI utility library</p></td></tr></tbody></table>

The runtime environment already has `dashscope`, `fastapi`, `uvicorn`, and `pyyaml` pre-installed; you may also pin versions in requirements.txt if needed.

### Step 2: Add observability code <span id="opd-tracing-step2" />

Code changes concentrate on 5 decorators/functions. They nest automatically to form a complete call chain:

```plaintext
# 调用链结构：装饰器自动嵌套形成 Span 树，[ENTRY] 为入口 Span，[LLM]/[TOOL]/[custom] 为子 Span
[ENTRY: ROLLOUT] @observe_processor          ← Rollout 处理器入口
├── [LLM] trace_client / @observe_llm        ← LLM 调用（学生模型调用）
│   └── (OpenAI / LangChain / DashScope API)
├── [TOOL] trace_tool / @observe_tool        ← 工具调用
│   ├── tool: my_tool (MCP)
│   └── tool: my_scorer (自定义)
└── [custom] rollout_metrics                 ← Rollout 自定义指标

[ENTRY: REWARD] @observe_processor           ← Reward 处理器入口
├── [LLM] trace_client / @observe_llm        ← LLM 调用（评分用）
├── [TOOL] trace_tool / @observe_tool        ← 工具调用
└── [custom] reward_metrics                  ← Reward 自定义指标
```

#### @observe\_processor — Track processor entry <span id="opd-tracing-observe-processor" />

Add it to the `process()` method to create the top-level ENTRY Span (the entry Span, the outermost layer of the call chain). The SDK infers the Span type from the inheritance chain: inheriting `AbstractRolloutProcessor` is recognized as ROLLOUT, and inheriting `AbstractRewardProcessor` is recognized as REWARD. It automatically records input, output, duration, and success/failure status.

<strong>Rollout side example (</strong>`functions/rollout/my_rollout.py`):

```plaintext
from dashscope.finetune.reinforcement.component.observability import observe_processor
# 导入 Rollout 处理器基类与输入输出类型（项目内已定义）

class MyRolloutProcessor(AbstractRolloutProcessor):
    @observe_processor  # Span 类型 = ROLLOUT（按继承 AbstractRolloutProcessor 自动推断）
    async def process(self, input: RolloutInput) -> RolloutOutput:
        # input: Rollout 输入，含 prompt、模型资源、工具列表等
        await self._async_setup()  # 异步初始化：建 LLM 客户端、加载工具
        return await self._async_process(input)  # 执行实际 Rollout 逻辑，返回轨迹与指标
```

<strong>Reward side example (</strong>`functions/reward/reward.py`):

```plaintext
class MyRewardProcessor(AbstractRewardProcessor):
    @observe_processor  # Span 类型 = REWARD（按继承 AbstractRewardProcessor 自动推断）
    async def process(self, input: RewardInput) -> RewardOutput:
        # input: Reward 输入，含 agent_output（模型轨迹）与 ground_truth（标准答案）
        messages = input.agent_output.messages  # 取模型多轮对话消息列表
        content = messages[-1].get("content", "") if messages else ""  # 取最后一条 assistant 回复文本
        score = await evaluate(content, input.ground_truth)  # 用标准答案对回复评分，返回主分
        return RewardOutput(
            reward=Reward(reward_score=score, reward_metrics={...}),  # 主分入 reward_score，分维度入 reward_metrics
            status=TaskStatus.SUCCESS,  # 任务状态：SUCCESS 表示评分成功
            error=None,  # 无异常时为 None；失败时填错误信息
        )
```

#### trace\_client() — Track LLM client <span id="opd-tracing-trace-client" />

Call it during initialization or in the process method to wrap the LLM client instance. Afterwards, all LLM requests from that client automatically produce an LLM Span that records the model name, request content, token usage, and latency.

<strong>Supported client types (auto-detected by object attributes):</strong>

- OpenAI clients (`AsyncOpenAI` / `OpenAI`)
- OpenAI completions resource (`.chat.completions`)
- LangChain ChatOpenAI and similar classes (via `.client` / `.async_client`)
- DashScope Generation class (pass the class itself, not an instance)

<strong>Example (</strong>`functions/rollout/my_rollout.py`):

```plaintext
from dashscope.finetune.reinforcement.component.observability import trace_client
# 导入 LLM 客户端包装函数，调用后该客户端的请求自动产生 LLM Span

class MyRolloutProcessor(AbstractRolloutProcessor):
    def _build_llm(self, input: RolloutInput) -> ChatOpenAI:
        # input: Rollout 输入，model_resource 含模型名等资源配置
        resource = input.model_resource  # 取模型资源（由 Runtime 注入的学生模型）
        llm = ChatOpenAI(model=resource.model_name, ...)  # 构造 LangChain ChatOpenAI 客户端
        trace_client(llm)  # 包装后自动追踪所有 LLM 调用（记录模型名、请求、Token 用量、延迟）
        return llm
```

#### trace\_tool() — Track tool calls <span id="opd-tracing-trace-tool" />

After obtaining a tool instance, call this to wrap the tool object. Afterwards, each tool call produces a TOOL Span that records the tool name, parameters, return value, and latency.

<strong>Supported input formats:</strong>

- A single LangChain BaseTool
- List / tuple / dict (auto-traversed)
- LangGraph ToolNode (auto-expands `.tools_by_name`)
- MCP tools (auto-detected, provider set to "mcp")
- Custom provider (`trace_tool(tool, provider="my-plugin")`)

<Warning>
  <strong>MCP note:</strong> The MCP (Model Context Protocol) Server and Client run in different processes. The Server-side `@observe_tool` is <strong>ineffective</strong> on the Client side. You must call `get_tools()` on the Client side first, then run `trace_tool(tools)` on the returned tool list.
</Warning>

#### @observe\_llm — Custom LLM function <span id="opd-tracing-observe-llm" />

When `trace_client()` cannot auto-detect your LLM client, use this decorator to manually mark an LLM call function.

<strong>Signature requirement:</strong> the function must include keyword arguments `model` and `messages` after `*`.

<strong>Example (</strong>`functions/rollout/my_rollout.py`):

```plaintext
from dashscope.finetune.reinforcement.component.observability import observe_llm
# 导入 LLM 装饰器，用于手动标记无法被 trace_client 自动识别的 LLM 调用

class MyRolloutProcessor(AbstractRolloutProcessor):
    @observe_llm  # 标记为 LLM Span（记录模型名、messages、Token 用量）
    async def _call_llm(self, *, messages: List[Dict], model: str) -> Any:
        # messages: 对话消息列表；model: 模型名（* 后关键字参数为签名硬性要求）
        ...
```

#### @observe\_tool — Custom tool function <span id="opd-tracing-observe-tool" />

When `trace_tool()` cannot auto-detect your tool (e.g., a plain Python function used as a tool), use this decorator to mark it manually. You can customize the Span name via the `name` parameter.

<strong>Example (</strong>`functions/rollout/my_rollout.py`):

```plaintext
from dashscope.finetune.reinforcement.component.observability import observe_tool
# 导入工具装饰器，用于手动标记无法被 trace_tool 自动识别的工具函数

class MyRolloutProcessor(AbstractRolloutProcessor):
    @observe_tool(name="my_scorer")  # 标记为 TOOL Span，name 自定义 Span 显示名
    def _score_response(self, *, messages: List[Dict]) -> float:
        # messages: 对话消息列表，函数返回评分为 float
        ...
```

### Step 3: Submit the task <span id="opd-tracing-step3" />

When submitting a task, leave the Runtime `env` field empty — Tracing is <strong>enabled by default</strong>:

```plaintext
# FunctionComponentRuntime: Rollout/Reward 组件运行时配置
rollout_runtime = FunctionComponentRuntime(
    cpu=2, memory_size=4096, disk_size=512,  # 资源：2 核 CPU、4GB 内存、512MB 磁盘
    concurrency=30, capacity=30,  # 并发 30、容量 30（单实例同时处理请求数）
    min_capacity=30, max_capacity=60,  # 弹性伸缩下限 30、上限 60
    env={}  # 留空 = 默认开启 Tracing；设 {"ENABLE_TRAJECTORY": "false"} 可关闭
)
```

<Note>
  <strong>To disable Tracing</strong> (to save cost), set `{"ENABLE_TRAJECTORY": "false"}` in `env`. After disabling, system metrics are unaffected; only Tracing data collection stops.
</Note>

## Tracing performance and cost trade-off <span id="opd-tracing-cost" />

### Data pipeline and cost sources

- <strong>Data flow</strong>: function code (decorators) → OpenTelemetry SDK → ARMS → console Trajectory/Metrics tabs
- <strong>Cost sources</strong>: ARMS Span storage (pay-as-you-go) + a small amount of CPU/network overhead on the function side + a slight increase in training latency

### Switch strategy and cost governance

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "20%" }} /><col style={{ width: "20%" }} /><col style={{ width: "20%" }} /><col style={{ width: "20%" }} /><col style={{ width: "20%" }} /></colgroup><thead><tr><th><p>Phase</p></th><th><p>Tracing status</p></th><th><p>Collected data</p></th><th><p>Not collected</p></th><th><p>Cost impact</p></th></tr></thead><tbody><tr><td><p>Development / small-batch debugging</p></td><td><p><strong>Fully on</strong></p></td><td><p>All (trajectories + Reward analysis + tool calls + system metrics)</p></td><td><p>—</p></td><td><p>Low</p></td></tr><tr><td><p>Canary / pre-release confirmation</p></td><td><p><strong>Retained</strong></p></td><td><p>All</p></td><td><p>—</p></td><td><p>Medium</p></td></tr><tr><td><p>Large-scale formal training</p></td><td><p><strong>Can disable</strong></p></td><td><p>actor/critic/trajectory/timing system metrics</p></td><td><p>Trajectory replay / tool call details / custom metrics curves</p></td><td><p>Significantly reduced</p></td></tr></tbody></table>

<strong>OPD difference:</strong> for system metrics retained after disabling Tracing, see §Training metrics reference; `trace/distillation/*` are system metrics and remain viewable after disabling (teacher model health troubleshooting is unaffected).

Cost governance for custom metrics:

- **Count and cardinality**: the count and cardinality of `reward_metrics` / `rollout_metrics` affect ARMS storage
- **Avoid high-cardinality fields**: do not use user\_id / request\_id as metric keys
- **Streamline metrics**: keep key metrics ≤10; do not leave temporary debugging metrics

## Custom metrics <span id="opd-custom-metrics" />

The key-value metrics defined through the following entry points in your code automatically appear in the `trace/` group of the console <strong>Metrics</strong> tab and on the <strong>Reward analysis</strong> page:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "33.333333%" }} /><col style={{ width: "33.333333%" }} /><col style={{ width: "33.333333%" }} /></colgroup><thead><tr><th><p><strong>Entry</strong></p></th><th><p><strong>Code location</strong></p></th><th><p><strong>Console path</strong></p></th></tr></thead><tbody><tr><td><p>reward\_metrics</p></td><td><p><code>Reward(reward\_metrics=\{"acc": 0.8})</code></p></td><td><p><code>trace/reward\_metrics/\{reward-name}/acc/\{avg,sum}</code></p></td></tr><tr><td><p>rollout\_metrics</p></td><td><p><code>AgentOutput(rollout\_metrics=\{"latency": 1.2})</code></p></td><td><p><code>trace/rollout\_metrics/latency/\{avg,sum}</code></p></td></tr></tbody></table>

<strong>Value type constraints:</strong>

- **Flat dict**: reward\_metrics / rollout\_metrics return a flat dict (not a nested list), with each value a float
- **Main score reported separately**: reward\_score goes into `Reward.reward_score` alone; do not duplicate it in reward\_metrics
- **Non-float only persisted**: non-float data is persisted only via `rollout_extra` and does not enter metrics
- **Metric path**: the reward\_score main score maps to `trace/reward/<name>/`, and each dimension maps to `trace/reward_metrics/<name>/<metric>`

<strong>Multiple Reward function scenario:</strong> use `RewardFunctionComponent(name="reward-1")` to give each Reward function a unique name; metric paths are distinguished automatically (`trace/reward_metrics/reward-1/...`).

## View effects in the console <span id="opd-console" />

After training starts, open the [Model Fine-tuning console](https://bailian.console.aliyun.com/cn-beijing/model/tuning), enter the <strong>Model Fine-tuning</strong> page, and click the task name to view observability data in the details. The following explains the correspondence between "what you did in code → what you see in the console".

<strong>OPD difference:</strong> the task details page has no standalone "Details" tab; status is viewed in the task list. Focus points: on the Metrics tab, watch the distillation loss, teacher model health, and validation reward; on the Trajectory tab, inspect tool call chains per sample; on the Output tab, the last Checkpoint is auto-published.

The task details contain the following tabs, to be used in the order "check progress first → then check behavior → drill down when issues arise":

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "33.333333%" }} /><col style={{ width: "33.333333%" }} /><col style={{ width: "33.333333%" }} /></colgroup><thead><tr><th><p>Tab</p></th><th><p>Main question answered</p></th><th><p>When to use</p></th></tr></thead><tbody><tr><td><p>Trajectory</p></td><td><p>What the model actually did, why the score is what it is</p></td><td><p>Verify model behavior, attribute low-score samples</p></td></tr><tr><td><p>Metrics</p></td><td><p>Quantitative training trends</p></td><td><p>Read curves to judge convergence, spot inflection points</p></td></tr><tr><td><p>Output</p></td><td><p>Checkpoint list and publishing</p></td><td><p>Select a model after training completes</p></td></tr><tr><td><p>Logs</p></td><td><p>stdout / stderr / error stack traces</p></td><td><p>Troubleshoot when FAILED</p></td></tr></tbody></table>

### Trajectory and behavior analysis <span id="opd-console-trajectory" />

<strong>Trace details — corresponds to @observe\_processor:</strong> on the <strong>Trajectory Details</strong> subpage of the <strong>Trajectory</strong> tab, you can see the complete interaction process of each Rollout:

- <strong>Trace list:</strong> shows all sampled trajectories; supports filtering by Sample ID / Trajectory ID / Epoch / Step
- <strong>Dialogue process:</strong> complete multi-turn interaction (user → assistant → tool\_call → tool\_result → assistant), intuitively showing the model's reasoning chain
- <strong>Reward score:</strong> each Step shows the corresponding Reward score and status (SUCCESS/FAILED)

Focus on: are the tool calls correct? are the conversation rounds reasonable? is the model repeating ineffective operations?

<span id="opd-console-tools" />

<strong>Tool calling analysis — corresponds to trace\_tool / @observe\_tool:</strong> on the <strong>Tool Calling Analysis</strong> subpage of the <strong>Trajectory</strong> tab, you can view:

- <strong>Tool call records:</strong> tool name, call parameters, return results, and duration
- <strong>Tracing sub-tab:</strong> the Span tree of each trajectory; expandable to view the full details of each tool call and LLM request

<strong>Typical use:</strong> troubleshoot an Agent's failed tool calls — which tool errored? were parameters passed correctly? was the duration too long?

<span id="opd-console-reward" />

<strong>Reward analysis — corresponds to reward\_metrics:</strong>

<strong>Concept definitions:</strong>

- <strong>Sample</strong>: one raw sample in the training data (one problem, one instruction, one prompt)
- <strong>Trajectory</strong>: one specific interaction trajectory produced by the same Sample under `n_rollouts` samplings
- Relationship: one Sample → N Trajectories (N = `n_rollouts`)

On the <strong>Reward analysis</strong> subpage of the <strong>Trajectory</strong> tab, evaluate training effects from three dimensions:

- <strong>Step dimension:</strong> select a training Step to view the Reward aggregation (average score, success rate, trend chart) of all samples under that Step, to judge overall training trend
- <strong>Sample dimension:</strong> select a Sample ID to compare the Reward of the same sample across different trajectories, to find problematic samples
- <strong>Trajectory dimension:</strong> view the raw score of each scoring dimension for a single trajectory, for attribution analysis

### Metrics, output, and logs <span id="opd-console-metrics" />

<strong>Metrics tab — corresponds to rollout\_metrics / reward\_metrics:</strong> in the `trace/` group of the <strong>Metrics</strong> tab, you can see the aggregation curves (avg / sum) of all custom metrics you defined in code. For the full metric groups, see §Training metrics reference below; for how to attribute an abnormal metric → §Judge training effects and troubleshoot.

<span id="opd-console-output" />

<strong>Output tab:</strong> after training completes, you can see the Checkpoint list in the <strong>Output</strong> tab; each row contains the Checkpoint ID, publish status, and remaining retention time.

1. Select the target Checkpoint and click the <strong>Publish</strong> button
2. Wait for publishing to complete (status changes from "Pending Publish" to "Published")
3. After publishing, you can call it via the model name in the API

<strong>OPD difference:</strong> the last Checkpoint is auto-published; the Output tab still allows manually selecting any Checkpoint to publish (it coexists with the auto-published last one; distinguish by model name when calling). Training completion does not equal training success. When Reward is enabled, it is recommended to use the Checkpoint with the best `validation/data/reward/mean@1` rather than the default last one. Pure distillation (without registering a reward component) has no validation reward; judge effects by loss trend and independent evaluation.

<span id="opd-console-logs" />

<strong>Logs tab:</strong> view training run logs on the <strong>Logs</strong> tab; they can also be obtained via the SDK / CLI:

- SDK: `AgenticRL.logs(job_id="ft-xxx", lines=100)`
- CLI: `dashscope rl logs "ft-xxx" --lines 100`

For FAILED task troubleshooting steps, see §FAQ → FAILED troubleshooting.

## Training metrics reference <span id="opd-metrics-ref" />

Below is the full set of OPD-specific metrics and common metrics. Metrics are displayed on the Metrics page; click to expand each group's details.

### OPD-specific metrics <span id="opd-metrics-opd" />

The following are OPD-specific metrics:

<Accordion title="distillation/ — distillation loss">
  <table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "50%" }} /><col style={{ width: "50%" }} /></colgroup><thead><tr><th><p><strong>Metric</strong></p></th><th><p><strong>Meaning and interpretation</strong></p></th></tr></thead><tbody><tr><td><p><code>actor/distillation/loss</code></p></td><td><p>Distillation loss; a decline indicates the student model is approaching the teacher model</p></td></tr></tbody></table>

  When reward is enabled, `actor/distillation/loss` and `actor/pg_loss` together constitute the total loss; both should decline or stabilize.
</Accordion>

<Accordion title="trace/distillation/ — teacher model signal health">
  <table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "50%" }} /><col style={{ width: "50%" }} /></colgroup><thead><tr><th><p><strong>Metric</strong></p></th><th><p><strong>Meaning and interpretation</strong></p></th></tr></thead><tbody><tr><td><p><code>trace/distillation/valid\_token\_rate</code></p></td><td><p>Proportion of tokens for which the teacher model successfully provided a logprob (log probability); normally close to 1.0; less than 1 indicates missing scores</p></td></tr><tr><td><p><code>trace/distillation/missing\_token\_rate</code></p></td><td><p>Proportion of tokens for which the teacher model did not provide a logprob; normally close to 0</p></td></tr><tr><td><p><code>trace/distillation/empty\_response\_rate</code></p></td><td><p>Proportion of empty responses returned by the teacher model; normally close to 0; an increase points to an anomaly on the teacher model side</p></td></tr></tbody></table>
</Accordion>

<Accordion title="validation/ — validation set metrics (reported when Reward is enabled)">
  <table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "50%" }} /><col style={{ width: "50%" }} /></colgroup><thead><tr><th><p><strong>Metric</strong></p></th><th><p><strong>Meaning and interpretation</strong></p></th></tr></thead><tbody><tr><td><p><code>validation/data/reward/mean\@1</code></p></td><td><p>Aggregated reward mean on the validation set; a rise relative to the baseline is healthy</p></td></tr></tbody></table>

  Validation metrics include only the aggregated reward mean. Custom dimensions returned by the Reward function are not reflected in validation metrics; view them under the training metric `trace/reward_metrics/<reward_name>/<metric>` (the dimension name comes from the keys of `reward_metrics` and varies by task). Pure distillation (without registering a reward component, functions=None) does not report this group.
</Accordion>

### Key metrics quick reference <span id="opd-metrics-quick" />

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "25%" }} /><col style={{ width: "40%" }} /><col style={{ width: "35%" }} /></colgroup><thead><tr><th><p><strong>Category</strong></p></th><th><p><strong>Metric</strong></p></th><th><p><strong>Interpretation</strong></p></th></tr></thead><tbody><tr><td><p>Distillation</p></td><td><p><code>actor/distillation/loss</code></p></td><td><p>Decline or stabilize</p></td></tr><tr><td><p>Teacher model health</p></td><td><p><code>trace/distillation/valid\_token\_rate</code></p></td><td><p>Close to 1.0</p></td></tr><tr><td><p>Teacher model health</p></td><td><p><code>trace/distillation/missing\_token\_rate</code></p></td><td><p>Close to 0</p></td></tr><tr><td><p>Teacher model health</p></td><td><p><code>trace/distillation/empty\_response\_rate</code></p></td><td><p>Close to 0</p></td></tr><tr><td><p>Validation</p></td><td><p><code>validation/data/reward/mean\@1</code></p></td><td><p>Rise relative to baseline</p></td></tr><tr><td><p>reward dimension</p></td><td><p><code>trace/reward\_metrics/\<reward\_name>/\<metric></code></p></td><td><p>Key business dimensions do not regress</p></td></tr></tbody></table>

### Common metrics <span id="opd-metrics-common" />

Metrics produced during training are grouped by prefix as follows; detailed metric descriptions for each group are in the collapsible panels below:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "25%" }} /><col style={{ width: "25%" }} /><col style={{ width: "25%" }} /><col style={{ width: "25%" }} /></colgroup><thead><tr><th><p><strong>Group prefix</strong></p></th><th><p><strong>Metric count</strong></p></th><th><p><strong>Type</strong></p></th><th><p><strong>Description</strong></p></th></tr></thead><tbody><tr><td><p><strong>actor/</strong></p></td><td><p>8</p></td><td><p>System</p></td><td><p>Policy network metrics: loss, entropy, KL divergence, clip ratio, gradient norm, learning rate</p></td></tr><tr><td><p><strong>critic/</strong></p></td><td><p>12</p></td><td><p>System</p></td><td><p>Reward and value estimation: mean / max / min of score / rewards / advantages / returns</p></td></tr><tr><td><p><strong>trajectory/</strong></p></td><td><p>15</p></td><td><p>System</p></td><td><p>Trajectory statistics: response length, prompt length, truncation rate, abort rate, conversation turns</p></td></tr><tr><td><p><strong>trace/</strong></p></td><td><p>40+</p></td><td><p><strong>Mixed</strong></p></td><td><p>Observability metrics: epoch, LLM call count, success rate, custom metrics</p></td></tr><tr><td><p><strong>timing/</strong></p></td><td><p>11</p></td><td><p>System</p></td><td><p>Latency analysis: Trainer phase duration, Rollout duration, per-token duration</p></td></tr></tbody></table>

<strong>OPD difference:</strong> OPD does not produce the `perf/` and `fully_async/` metric groups.

Click to expand the full metric list for each group:

<Accordion title="actor/ — policy network metrics (8)">
  <table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "50%" }} /><col style={{ width: "50%" }} /></colgroup><thead><tr><th><p><strong>Metric</strong></p></th><th><p><strong>Meaning</strong></p></th></tr></thead><tbody><tr><td><p><code>actor/loss</code></p></td><td><p>Total loss (pg + entropy + ...)</p></td></tr><tr><td><p><code>actor/pg\_loss</code></p></td><td><p>Policy gradient loss</p></td></tr><tr><td><p><code>actor/entropy</code></p></td><td><p>Average token entropy of the current policy, reflecting exploration level</p></td></tr><tr><td><p><code>actor/ppo\_kl</code></p></td><td><p>KL divergence of the current policy relative to the initial policy</p></td></tr><tr><td><p><code>actor/pg\_clipfrac</code></p></td><td><p>Importance sampling clip trigger rate, reflecting policy drift speed</p></td></tr><tr><td><p><code>actor/pg\_clipfrac\_lower</code></p></td><td><p>Dual-clip lower-bound clip trigger rate (always 0 when dual-clip is not enabled)</p></td></tr><tr><td><p><code>actor/grad\_norm</code></p></td><td><p>Gradient norm</p></td></tr><tr><td><p><code>actor/lr</code></p></td><td><p>Current learning rate</p></td></tr></tbody></table>
</Accordion>

<Accordion title="critic/ — reward and value estimation (12)">
  <table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "50%" }} /><col style={{ width: "50%" }} /></colgroup><thead><tr><th><p><strong>Metric</strong></p></th><th><p><strong>Meaning</strong></p></th></tr></thead><tbody><tr><td><p><code>critic/score/\{mean,max,min}</code></p></td><td><p>Raw reward score statistics (before KL deduction)</p></td></tr><tr><td><p><code>critic/rewards/\{mean,max,min}</code></p></td><td><p>Final training Reward statistics after KL penalty deduction</p></td></tr><tr><td><p><code>critic/advantages/\{mean,max,min}</code></p></td><td><p>Advantage function statistics, reflecting the current policy's improvement over the baseline</p></td></tr><tr><td><p><code>critic/returns/\{mean,max,min}</code></p></td><td><p>Returns (Critic target) statistics</p></td></tr></tbody></table>
</Accordion>

<Accordion title="trajectory/ — trajectory statistics (15)">
  <table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "50%" }} /><col style={{ width: "50%" }} /></colgroup><thead><tr><th><p><strong>Metric</strong></p></th><th><p><strong>Meaning</strong></p></th></tr></thead><tbody><tr><td><p><code>trajectory/response\_length/\{mean,max,min}</code></p></td><td><p>Response token count statistics (including abort samples)</p></td></tr><tr><td><p><code>trajectory/response/aborted\_ratio</code></p></td><td><p>Proportion of trajectories with zero response length</p></td></tr><tr><td><p><code>trajectory/response\_length\_non\_aborted/\{mean,max,min}</code></p></td><td><p>Effective response token count statistics after excluding aborts</p></td></tr><tr><td><p><code>trajectory/response\_length/clip\_ratio</code></p></td><td><p>Proportion of responses truncated at max length</p></td></tr><tr><td><p><code>trajectory/prompt\_length/\{mean,max,min}</code></p></td><td><p>Prompt token count statistics</p></td></tr><tr><td><p><code>trajectory/prompt\_length/clip\_ratio</code></p></td><td><p>Proportion of prompts truncated at max length</p></td></tr><tr><td><p><code>trajectory/num\_turns/\{mean,max,min}</code></p></td><td><p>Statistics of turns of interaction between the Agent and the LLM</p></td></tr></tbody></table>
</Accordion>

<Accordion title="trace/ — observability metrics">
  <table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "50%" }} /><col style={{ width: "50%" }} /></colgroup><thead><tr><th><p><strong>Metric</strong></p></th><th><p><strong>Meaning</strong></p></th></tr></thead><tbody><tr><td><p><code>trace/training/epoch</code></p></td><td><p>Current training epoch</p></td></tr><tr><td><p><code>trace/num\_llm\_calls/\{avg,sum}</code></p></td><td><p>Per-trajectory / total LLM call count</p></td></tr><tr><td><p><code>trace/success\_rate/agent/\{avg,sum}</code></p></td><td><p>Agent task success rate / cumulative success count</p></td></tr><tr><td><p><code>trace/success\_rate/reward/\{avg,sum}</code></p></td><td><p>Reward computation success rate / cumulative success count</p></td></tr><tr><td><p><code>trace/attempts/agent/\{avg,sum}</code></p></td><td><p>Agent average / cumulative HTTP attempt count (including retries)</p></td></tr><tr><td><p><code>trace/reward/\<reward\_name>/\{avg,sum}</code></p></td><td><p>Average / cumulative value of a single Reward function</p></td></tr><tr><td><p><code>trace/reward\_metrics/\<reward\_name>/\<metric>/...</code></p></td><td><p>Custom sub-metrics returned by a Reward function</p></td></tr></tbody></table>
</Accordion>

<Accordion title="timing/ — latency analysis (11)">
  Divided into three subgroups: `timing/s/*` (Trainer phase duration, seconds), `timing/s/rollout/*` (Rollout-side duration, seconds), and `timing/ms/*_per_token` (per-token duration, milliseconds).

  <table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "50%" }} /><col style={{ width: "50%" }} /></colgroup><thead><tr><th><p><strong>Metric</strong></p></th><th><p><strong>Meaning</strong></p></th></tr></thead><tbody><tr><td><p><code>timing/s/step</code></p></td><td><p>Total duration of a complete Trainer step</p></td></tr><tr><td><p><code>timing/s/trainer\_fetch\_batch</code></p></td><td><p>Time for the Trainer to wait for and fetch a batch</p></td></tr><tr><td><p><code>timing/s/old\_log\_prob</code></p></td><td><p>Log-prob computation time of the old policy</p></td></tr><tr><td><p><code>timing/s/adv</code></p></td><td><p>Advantage computation time</p></td></tr><tr><td><p><code>timing/s/update\_actor</code></p></td><td><p>Actor backpropagation + optimizer step time</p></td></tr><tr><td><p><code>timing/s/param\_sync</code></p></td><td><p>Time for the Rollouter to sync parameters from the Trainer</p></td></tr><tr><td><p><code>timing/s/rollout/agent\_loop\_latency/avg</code></p></td><td><p>Total duration of a single Rollout</p></td></tr><tr><td><p><code>timing/s/rollout/model\_latency/avg</code></p></td><td><p>Cumulative LLM inference time for a single trajectory</p></td></tr><tr><td><p><code>timing/s/rollout/reward\_latency/avg</code></p></td><td><p>Reward call duration</p></td></tr><tr><td><p><code>timing/ms/gen\_per\_token</code></p></td><td><p>Per-token duration in the generation phase</p></td></tr><tr><td><p><code>timing/ms/update\_actor\_per\_token</code></p></td><td><p>Per-token duration in the Actor update phase</p></td></tr></tbody></table>
</Accordion>

## Judge training effects <span id="opd-judge" />

The metrics reported during training are sufficient to judge whether a round of OPD is converging normally; post-training business effect evaluation is performed by the user according to their own criteria.

### In-training interpretation order <span id="opd-judge-in-training" />

On the Metrics page, interpret in the following order:

<strong>① Teacher model signal completeness</strong> — `trace/distillation/valid_token_rate` is close to 1.0. When noticeably below 1, teacher model scoring is missing, the explanatory power of other metrics decreases, and you should troubleshoot first (troubleshooting decision tree P2).

<strong>② Distillation convergence</strong> — `actor/distillation/loss` declines or stabilizes over training, indicating the student model is approaching the teacher model.

<strong>③ Validation set performance</strong> — `validation/data/reward/mean@1` rises relative to the pre-training baseline. View per-dimension via `trace/reward_metrics/<reward_name>/<metric>`; key business dimensions (especially safety) must not regress.

Validation frequency is controlled by `eval_steps`. If a validation point exists at step 0, that point corresponds to the pre-training model state and can be used directly as the baseline.

Pure distillation (without registering a reward component, functions=None) produces no validation metrics or per-dimension metrics. Judge whether training is normal based on the distillation loss and teacher model health metrics; judge business effects via post-training evaluation.

### Post-training business evaluation <span id="opd-judge-post" />

<strong>Keep the pre-training baseline</strong> — distillation gain is a relative quantity; without a baseline, the improvement magnitude cannot be judged. Training produces a new model ID; the original model remains callable, but you must retain the baseline evaluation results yourself.

<strong>Maintain consistent criteria</strong> — pre- and post-training evaluation should use the same evaluation set and decoding parameters (`temperature`, `max_tokens`, whether thinking is enabled, etc.). When criteria are inconsistent, the difference is not comparable — for example, a response truncated by the length limit can manifest as a false decline in capability.

The example directory ships with evaluation and comparison scripts; these are only implementation references and can be modified as needed.

### Acceptance criteria <span id="opd-judge-criteria" />

Four items must be satisfied simultaneously:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "25%" }} /><col style={{ width: "75%" }} /></colgroup><thead><tr><th><p><strong>Criterion</strong></p></th><th><p><strong>Requirement</strong></p></th></tr></thead><tbody><tr><td><p>Main criterion</p></td><td><p>Core evaluation metric improves and is not random fluctuation</p></td></tr><tr><td><p>Safety criterion</p></td><td><p>Safety dimensions are no higher than before training</p></td></tr><tr><td><p>Format criterion</p></td><td><p>Format dimensions are no lower than before training</p></td></tr><tr><td><p>Per-category criterion</p></td><td><p>No category significantly regresses after splitting by business category</p></td></tr></tbody></table>

<Accordion title="Measured value reference (for reference only, not a universal criterion)">
  Example three-way comparison (200 validation samples, temperature=0, 8 concurrency):

  <table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "40%" }} /><col style={{ width: "20%" }} /><col style={{ width: "20%" }} /><col style={{ width: "20%" }} /></colgroup><thead><tr><th><p><strong>Metric</strong></p></th><th><p><strong>Base</strong></p></th><th><p><strong>Teacher</strong></p></th><th><p><strong>OPD</strong></p></th></tr></thead><tbody><tr><td><p>Core success rate</p></td><td><p>0.700</p></td><td><p>0.955</p></td><td><p>0.990</p></td></tr><tr><td><p>Safety violation rate</p></td><td><p>0.235</p></td><td><p>0</p></td><td><p>0</p></td></tr></tbody></table>

  Why the student model surpasses the teacher model: when reward is enabled, reward scores against the reference answer, tending toward the reference answer rather than the teacher model; only pure distillation takes the teacher model as the performance ceiling.
</Accordion>

## OPD troubleshooting decision tree <span id="opd-troubleshooting" />

Spot an abnormal curve on the Metrics page → locate the issue against this section → handle per the recommendations.

### P1 Distillation loss not declining <span id="opd-ts-p1" />

- <strong>Main signal</strong>: `actor/distillation/loss` does not decline over training or even rises
- <strong>Root cause</strong>: teacher model signal missing / learning rate unsuitable (too small to converge or too large to break the output format) / poor data quality
- <strong>Handling</strong>: first check whether `valid_token_rate` is close to 1.0 → when the signal is normal, loss not declining is often due to a too-small learning rate; increase `learning_rate` (×1.5\~2, see the training config tuning decision table) → spot-check the reference answers in `rollout_extra` of the training data

### P2 Teacher model scoring missing <span id="opd-ts-p2" />

- <strong>Main signal</strong>: `trace/distillation/valid_token_rate` is noticeably below 1.0
- <strong>Root cause</strong>: scoring missing on the teacher model side (the teacher model is hosted on the server, opaque to the user, and cannot be troubleshot directly)
- <strong>Handling</strong>: check whether `missing_token_rate` and `empty_response_rate` rise in tandem → if the teacher model timeout or empty-response ratio is high, submit a ticket

### P3 Validation set not rising <span id="opd-ts-p3" />

- <strong>Main signal</strong>: `validation/data/reward/mean@1` shows a downward inflection while training reward keeps rising
- <strong>General handling</strong>: early stop (select an earlier Checkpoint on the Output tab) + keep pre- and post-training evaluation criteria consistent + view `reward_metrics` regression per dimension
- <strong>Additional root cause</strong>: overfitting the teacher model style (in pure distillation)
- <strong>Additional handling</strong>: spot-check the reference answer quality in `rollout_extra`

<span id="opd-ts-four-source" />

<strong>Four-source collaborative troubleshooting:</strong> the four types of observation sources each answer different questions: <strong>Logs</strong> → "when did it fail, error stack", <strong>Metrics</strong> → "is it happening / how severe", <strong>Trajectory</strong> → "why / which sample", <strong>Trajectory page · Tracing sub-tab</strong> → "which segment is slow / which external dependency is abnormal". Select the observation source by problem type:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "25%" }} /><col style={{ width: "25%" }} /><col style={{ width: "25%" }} /><col style={{ width: "25%" }} /></colgroup><thead><tr><th><p><strong>Problem type</strong></p></th><th><p><strong>Primary</strong></p></th><th><p><strong>Secondary</strong></p></th><th><p><strong>Not needed</strong></p></th></tr></thead><tbody><tr><td><p>Training divergence</p></td><td><p>Metrics</p></td><td><p>Trajectory</p></td><td><p>Logs</p></td></tr><tr><td><p>Task FAILED</p></td><td><p>Logs</p></td><td><p>Metrics</p></td><td><p>Trajectory</p></td></tr><tr><td><p>Reward not rising</p></td><td><p>Metrics → Trajectory</p></td><td><p>reward\_metrics</p></td><td><p>Logs</p></td></tr><tr><td><p>Tool call error</p></td><td><p>Tracing sub-tab</p></td><td><p>Logs</p></td><td><p>—</p></td></tr><tr><td><p>Training slowdown</p></td><td><p>Metrics timing</p></td><td><p>Tracing sub-tab</p></td><td><p>—</p></td></tr><tr><td><p>Poor output quality</p></td><td><p>Trajectory</p></td><td><p>reward\_metrics</p></td><td><p>—</p></td></tr></tbody></table>

<strong>OPD-specific problem types and source selection:</strong>

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "25%" }} /><col style={{ width: "25%" }} /><col style={{ width: "25%" }} /><col style={{ width: "25%" }} /></colgroup><thead><tr><th><p><strong>Problem type</strong></p></th><th><p><strong>Primary</strong></p></th><th><p><strong>Secondary</strong></p></th><th><p><strong>Not needed</strong></p></th></tr></thead><tbody><tr><td><p>Distillation loss not declining</p></td><td><p>Metrics</p></td><td><p>Trajectory</p></td><td><p>Logs</p></td></tr><tr><td><p>Teacher model scoring missing</p></td><td><p>Metrics</p></td><td><p>Logs</p></td><td><p>Trajectory</p></td></tr><tr><td><p>Validation set not rising</p></td><td><p>Metrics → Trajectory</p></td><td><p>reward\_metrics</p></td><td><p>Logs</p></td></tr></tbody></table>

## Next steps <span id="opd-next-steps" />

- [OPD Training Overview](opd-training-overview.md) → principles and end-to-end flow
- [Model OPD Development](opd-model-development-guide.md) / [Agentic OPD Development](opd-agentic-development-guide.md) → full function development details
- [OPD Training Configuration](opd-training-config.md) → full set of submission parameters and hyperparameters

## FAQ <span id="opd-faq" />

### Teacher model side anomalies <span id="opd-faq-teacher" />

The teacher model is deployed on the server and is opaque to the user. The following symptoms require submitting a ticket:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "60%" }} /><col style={{ width: "40%" }} /></colgroup><thead><tr><th><p><strong>Symptom</strong></p></th><th><p><strong>Handling</strong></p></th></tr></thead><tbody><tr><td><p><code>trace/distillation/valid\_token\_rate</code> \< 1, teacher model scoring missing</p></td><td><p>Submit a ticket</p></td></tr><tr><td><p>Teacher model timeout or scoring missing</p></td><td><p>Submit a ticket</p></td></tr><tr><td><p>Same-family teacher/student still reports a tokenizer mismatch</p></td><td><p>Submit a ticket</p></td></tr></tbody></table>

### Self-check items <span id="opd-faq-self" />

General self-check items:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "60%" }} /><col style={{ width: "40%" }} /></colgroup><thead><tr><th><p><strong>Symptom</strong></p></th><th><p><strong>Troubleshooting direction</strong></p></th></tr></thead><tbody><tr><td><p>Tracing console is blank</p></td><td><p>①Confirm env is not set to <code>ENABLE\_TRAJECTORY="false"</code> ②Confirm ARMS is authorized ③Check requirements.txt contains OTel dependencies ④Check process() has @observe\_processor</p></td></tr><tr><td><p>Reward intermittent FAILED</p></td><td><p>Unhandled empty messages / encoding errors → add try/except returning <code>TaskStatus.FAILED + error</code></p></td></tr><tr><td><p>Low format-type dimension</p></td><td><p>Check whether the student model output format meets the Reward function requirements</p></td></tr><tr><td><p>How to distinguish different Reward functions</p></td><td><p>Set a unique name via <code>RewardFunctionComponent(name="reward-1")</code>; metric paths are distinguished automatically</p></td></tr></tbody></table>

OPD-specific self-check items:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "60%" }} /><col style={{ width: "40%" }} /></colgroup><thead><tr><th><p><strong>Symptom</strong></p></th><th><p><strong>Troubleshooting direction</strong></p></th></tr></thead><tbody><tr><td><p>400 ... teacher\_model must be a valid model ID</p></td><td><p>Use a Model Studio model ID</p></td></tr><tr><td><p>On submission, error says the SDK lacks OPD capability</p></td><td><p>Switch to the wheel specified in the example</p></td></tr><tr><td><p>Pure distillation has no validation metrics</p></td><td><p>Normal behavior (no reward component registered, functions=None); judge via independent evaluation</p></td></tr><tr><td><p>format\_valid abnormally low</p></td><td><p>Check whether the student model output is polluted by thinking/tool-call text, and confirm the Reward format-judgment regex matches the pre- and post-training decoding parameters</p></td></tr></tbody></table>

### Fault quick reference and FAILED troubleshooting <span id="opd-faq-quick" />

General fault quick reference:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "40%" }} /><col style={{ width: "30%" }} /><col style={{ width: "30%" }} /></colgroup><thead><tr><th><p><strong>Symptom</strong></p></th><th><p><strong>First check</strong></p></th><th><p><strong>Next check</strong></p></th></tr></thead><tbody><tr><td><p>FAILED · function registration failed</p></td><td><p>classpath wrong / dependency missing</p></td><td><p>Check requirements.txt</p></td></tr><tr><td><p>FAILED · Rollout timeout</p></td><td><p>Single-traj timeout</p></td><td><p>timeout ↑ / use Tracing to find the slow segment</p></td></tr><tr><td><p>Tracing not visible</p></td><td><p>Console blank</p></td><td><p>Check ARMS authorization / requirements.txt</p></td></tr></tbody></table>

OPD-specific fault quick reference:

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "40%" }} /><col style={{ width: "30%" }} /><col style={{ width: "30%" }} /></colgroup><thead><tr><th><p><strong>Symptom</strong></p></th><th><p><strong>First check</strong></p></th><th><p><strong>Next check</strong></p></th></tr></thead><tbody><tr><td><p>valid\_token\_rate low</p></td><td><p>Teacher model side anomaly</p></td><td><p>Submit a ticket</p></td></tr><tr><td><p>Distillation loss not declining</p></td><td><p>Teacher model signal completeness</p></td><td><p>Learning rate / data quality</p></td></tr><tr><td><p>tool\_call\_count is 0 / NO\_TOOL frequent</p></td><td><p>Whether the student model supports function calling, and whether the passed tools parameter is correct</p></td><td><p>Check tool definition and api\_key/base\_url</p></td></tr></tbody></table>

<span id="opd-faq-failed" />

<strong>Standard troubleshooting flow:</strong>

1. <strong>Step 1 Task list</strong> confirm status and the failure time point
2. <strong>Step 2 Logs tab</strong> last 100-500 lines (SDK `AgenticRL.logs(job_id, lines=100)` / CLI `dashscope rl logs --lines 100`)
3. <strong>Step 3 Distinguish error layer</strong>:

   - <strong>User function error</strong> (Rollout/Reward throws an exception) → `test_functions` to reproduce locally → fix code → re-register/run
   - <strong>Framework error</strong> (OOM / insufficient resources / network) → tune `concurrency` / `capacity`
   - <strong>Data error</strong> (JSONL parse failed) → validate single-line format and `rollout_extra`

<strong>Common error patterns during training:</strong>

<table style={{ display: "table", tableLayout: "fixed", width: "100%" }}><colgroup><col style={{ width: "33.333333%" }} /><col style={{ width: "33.333333%" }} /><col style={{ width: "33.333333%" }} /></colgroup><thead><tr><th><p>Error pattern</p></th><th><p>Main symptom</p></th><th><p>Handling suggestion</p></th></tr></thead><tbody><tr><td><p>Insufficient resources</p></td><td><p>Scaling cannot keep up</p></td><td><p>See Step 3 framework error (tune <code>concurrency</code> / <code>capacity</code>)</p></td></tr><tr><td><p>Data format error</p></td><td><p>JSONL parse failed</p></td><td><p>Line-level JSON validation / <code>messages</code> roles / <code>rollout\_extra</code> field</p></td></tr></tbody></table>