Reinforcement learning training

• Reinforcement Learning Development Guide
• Reinforcement Learning Training Configuration
• Observability Configuration for Reinforcement Learning

​

RL training workflow

1. Set up the environment — Install the offline SDK, get an API key, and complete RL service authorization.
2. Prepare data and code — Download the demo package to understand the data format and project structure.
3. Submit the training job — Run the demo to deploy functions, upload data, and submit the job in one step.
4. Monitor training status — View the training status, logs, and reward trends.
5. Publish and deploy the model — Publish a checkpoint to "My Models" and deploy it for API access.

​

How RL training works

Training data (user questions + reference answers)
            │
            ▼
Rollout: The model generates candidate results based on the current policy.
            │
            ▼
Reward: Scores the result on a scale of 0 to 1 by comparing it with the reference answer. This step can be merged with the rollout.
            │
            ▼
Policy update: Reinforce high-scoring policies and suppress low-scoring ones based on the reward signal.
            │
            ▼
Iteration: Resample using the updated model to improve quality in each round.

​

Supported models

​

Training unit pricing

​

Environment setup

1. Get an API key: Obtain a DashScope API key from the Model Studio console.
2. Authorize the RL service: On the Model Fine-tuning page in the Model Studio console, complete the one-click authorization. The first time you use RL training, the console prompts you to authorize Alibaba Cloud OpenTelemetry, Function Compute (FC), and Log Service (SLS).
3. Set environment variables: The local SDK reads the following variables to control packaging, uploading, and deployment:
   • DASHSCOPE_API_KEY (Required): Your API key.
   • FC_PYPI_LIB (Required): Specifies the offline dashscope package to be installed when the FC container starts. The SDK automatically injects this variable as FC_SDK_PACKAGE during deployment, so you do not need to install it manually in FC. The .whl filename must exactly match this variable and be placed in the project's root directory.
   • LOG_LEVEL (Optional, default: info): Set to debug to output complete request/response information, which is useful for troubleshooting submission failures.

export DASHSCOPE_API_KEY="sk-your-api-key"
export FC_PYPI_LIB="dashscope-1.25.23-py3-none-any.whl"
export LOG_LEVEL="info"

​

Data and code preparation

1. Download the demo package agentic-rl-example.zip. (This demo requires a minimum of 24 Type IV model training units to run.)
2. After downloading the demo package, install the DashScope SDK (requires Python >= 3.10). Follow the instructions in the demo package to install the required dependencies:

# Install the SDK locally from the PyPI source. You do not need to install it in the FC container, which automatically installs the offline .whl package specified by FC_PYPI_LIB:
pip install dashscope

agentic-rl-example/
├── submit_job.py                         ← Main entry point for submitting the training job
├── test_functions.py                     ← Function registration and remote testing
├── config.yaml                           ← Optional YAML configuration file
├── job.yaml                              ← YAML submission configuration (CLI run / SDK init)
├── cli.sh                                ← Collection of local testing and query scripts
├── dashscope-1.25.23-py3-none-any.whl   ← Offline SDK installation package
├── requirements.txt                      ← List of dependencies
├── data/
│   ├── calc_train.jsonl                  ← Training data (full version)
│   ├── calc_train_min.jsonl              ← Training data (demo uses the minimal version by default)
│   ├── calc_validation.jsonl             ← Validation data (full version)
│   └── calc_validation_min.jsonl         ← Validation data (demo uses the minimal version by default)
├── functions/
│   ├── __init__.py
│   ├── rollout/
│   │   ├── rollout.py                    ← Rollout function (LangGraph + MCP tool)
│   │   └── rollout_only.py              ← Rollout function (simple version, with trajectory example)
│   └── reward/
│       ├── reward.py                     ← Reward function (basic scoring)
│       ├── reward_decorator.py           ← Reward function (decorator style)
│       └── group_reward.py              ← Reward function (group scoring)
├── resources/
│   ├── rollout_input.json               ← Rollout test input
│   ├── reward_input.json                ← Reward test input
│   ├── reward_decorator_input.json      ← Decorator reward test input
│   └── group_reward_input.json          ← Group reward test input
└── scripts/                             ← Local startup and query helper scripts (build/install/start_local_*/query_local_*/query_online_tuning/convert_parquet_to_jsonl)

​

Demo

• Rollout function: Calls the model being trained, using a calculator tool to answer math problems.
• Reward function: Compares the model's output with the reference answer to determine if the response is correct and assigns a score.

​

Data format

{"messages": [{"content": "Output the answer when you are ready. The answer should be surrounded by three sharps (`###`), in the form of ### ANSWER: <answer> ###. 6.6 minus x (3/2) times equals 5.6.", "role": "user"}], "rollout_extra": {"solution": "2/3"}}

• messages: The user's question, in a format consistent with ChatML.
• rollout_extra: Contains the reference answer or other business data, which is passed to the reward function for scoring. You can customize the fields in this object.

​

Recommended dataset size

• Getting started: A few dozen to a few hundred entries are enough to validate your approach (the data volume must be greater than the batch_size).
• Production training: A larger dataset generally yields better results. Large datasets also support a larger batch size, which helps improve training stability.

​

Job submission

python submit_job.py

import asyncio
from dashscope.finetune.reinforcement import (
    RolloutFunctionComponent, RewardFunctionComponent,
    FunctionComponentModel, FunctionComponentRuntime,
    TrainingDataset, ValidationDataset, DataSourceType,
)
from dashscope.finetune.agentic_rl import AgenticRL

async def main():
    client = AgenticRL()

    # client.run() handles everything in one step: register functions to FC → upload training data → submit the training job
    result = await client.run(
        model="qwen3.5-9b",                              # Base model
        training_datasets=[TrainingDataset(              # Training dataset
            data_source_type=DataSourceType.FILE_ID,
            file_name="./data/calc_train_min.jsonl",
        )],
        validation_datasets=[ValidationDataset(          # Validation dataset (optional)
            data_source_type=DataSourceType.FILE_ID,
            file_name="./data/calc_validation_min.jsonl",
        )],
        functions=[
            # Rollout function: defines how the model performs policy sampling
            RolloutFunctionComponent(
                name="rollout-1",
                timeout=600,
                fcmodel=FunctionComponentModel(
                    classpath="functions.rollout.rollout.CalcXRolloutProcessor"),
                runtime=FunctionComponentRuntime(
                    cpu=2, memory_size=4096, disk_size=512,
                    concurrency=30, capacity=30,
                    min_capacity=30, max_capacity=60,
                    memory_scale_threshold=0.6,
                    concurrency_scale_threshold=0.6,
                    env={})),                            # FC function environment variables
            RewardFunctionComponent(
                name="reward-1",
                weight=1.0,
                timeout=120,
                reward_metric_weight={"accuracy": 0.7, "format": 0.3},  # Keys are the metric names returned by the reward function
                fcmodel=FunctionComponentModel(
                    classpath="functions.reward.reward.DemoRewardProcessor"),
                runtime=FunctionComponentRuntime(
                    cpu=2, memory_size=4096, disk_size=512,
                    concurrency=30, capacity=30,
                    min_capacity=30, max_capacity=60,
                    memory_scale_threshold=0.6,
                    concurrency_scale_threshold=0.6,
                    env={})),
        ],
        # MTU billing configuration
        resources={
            "charge_type": "mtu_postpaid",   # Pay-as-you-go
            "mtu_spec_code": "MTU4",         # Training unit specification
            "mtu_capacity": 24,              # Number of MTUs
        },
        # Hyperparameters for the GSPO algorithm (the following are all required hyperparameters for the non-MoE qwen3.5-9b model)
        hyper_parameters={
            "algorithm": "gspo",             # Training algorithm
            "batch_size": 64,                # Batch size
            "eval_steps": 1,                 # Perform evaluation every N steps
            "kl_loss_coef": 0.002,           # KL divergence loss coefficient
            "learning_rate": 2e-6,           # Learning rate
            "lr_scheduler_type": "linear",   # Learning rate scheduler policy
            "max_length": 8192,              # Maximum sequence length (prompt + response)
            "n_epochs": 1,                   # Number of training epochs
            "n_rollouts": 8,                 # Number of rollouts per sample
            "ppo_mini_batch_size": 8,        # PPO mini-batch size
            "save_strategy": "epoch",        # Save strategy (steps or epoch)
        })

    if result.status_code == 200:
        print(f"Training job submitted. Job ID: {result.output.job_id}")
    else:
        print(f"Submission failed: {result}")

asyncio.run(main())

​

Getting started tips

• Start with a minimal dataset: The demo defaults to using calc_train_min.jsonl and n_epochs=1. Run this once to confirm the entire pipeline works before switching to the full dataset. This helps you avoid wasting 24 MTUs on a large dataset only to discover a configuration error.
• Do not change the hyperparameters initially: The 11 hyperparameters listed above are the recommended starting point for qwen3.5-9b. Do not adjust them for your first training run. When you do make adjustments, change only one variable at a time (like the learning rate or batch size) and run for at least eval_steps × 3 steps before evaluating the trend.
• Focus on one core metric: On the Metrics tab, watch critic/rewards/mean. A steady increase indicates the model is learning. If it stagnates or decreases, stop the job and investigate instead of letting it run idly.
• If the job fails, check the logs first: If a job status is FAILED, go to the Logs tab and check the error messages at the end, or use the SDK to pull the logs: AgenticRL.logs(job_id="ft-xxx", lines=100). Configuration or dependency errors are usually easy to spot.

​

Training monitoring and analysis

​

Model publishing and deployment

​

Next steps

• Learn the full details of function development → Reinforcement Learning Development Guide
• Learn about training configuration and monitoring → Reinforcement Learning Training Configuration — Submission and Configuration
• Learn about observability configuration, trajectory viewing, and training metrics → Observability Configuration and Metric Reference for Reinforcement Learning