Evaluating models across multiple dimensions using datasets helps you fully understand their performance and ensures they deliver effective results in real-world applications.
Getting Started
What is model evaluation
Model evaluation is a model quality validation tool provided by the Model Studio platform. It helps you objectively and quantitatively assess large language model performance.
One-sentence summary: Give the model a test, score responses automatically or manually, then generate an evaluation report.
Why perform model evaluation
- Scenario 1: Model selection decisions
- Scenario 2: Validate fine-tuning effectiveness
- Scenario 3: Quantify model capabilities
- Scenario 4: Continuously monitor model performance
- Prepare 100 test cases from your business scenarios.
- Evaluate all candidate models using the same dataset.
- Compare evaluation reports (scores, pass rates, sample performance).
- Select the model that best meets your business requirements.
- Avoid subjective judgments—let data drive decisions.
- Save time compared to manual testing of each model.
- Reduce risks of poor model selection.
Core value summary
Value | Description |
|---|---|
Objective quantification | Replace subjective judgments with data—avoid vague descriptions like "seems okay" |
Efficient comparison | Quickly compare multiple models—save manual testing time |
Risk governance | Thoroughly validate before deployment—reduce model selection risks |
Continuous optimization | Provide quantitative feedback for model fine-tuning and iteration |
Decision support | Provide data support for team collaboration and management reporting |
Evaluation workflow
Phase | Sub-step description |
|---|---|
1. Preparations | Evaluation dimensions (define scoring criteria) → Test data (prepare questions and answers) |
2. Create evaluation task | Select model → Select data → Select dimensions → Choose whether to join leaderboard |
3. Execute evaluation | Automatic evaluation (system completes automatically) or manual evaluation (requires manual annotation) |
4. View results | Score report → Pass rate statistics → Detailed breakdown |
Quick experience
Evaluation scenario: Evaluate Qwen-Max accuracy on common-sense question answering.
Complete example: Use 10 test questions to assess semantic similarity of model responses.
Experience steps:
- 1. Create evaluation dimension
- 2. Prepare test data
- 3. Create evaluation task
- 4. Check task status
Item | Content |
|---|---|
Dimension Name | Common-sense QA accuracy |
Description | Evaluate model accuracy in answering common-sense questions |
Type | Select Model Evaluation - Numeric |
Judge Model | Select Qwen-Max |
Scorer Template | Select Semantic Similarity |
Scoring Range | Keep default 0~5 |
Pass Threshold | 3.0 |
Template | Description | Applicable scenarios | Need custom Prompt? |
|---|---|---|---|
Comprehensive Evaluation (default) | A system preset template that scores across five dimensions: relevance, cultural sensitivity, information richness, clarity, and user engagement. | General conversations and customer service scenarios | No—use directly or modify as needed |
Semantic Similarity | A system preset template that evaluates the semantic closeness between the output and the reference answer. | QA systems and knowledge retrieval | No—use directly or modify as needed |
Custom Evaluation | Write your own prompt. | Specific business scenarios | Yes—write according to your requirements |
Core concepts
Custom evaluation vs. baseline evaluation comparison
Model Studio model evaluation supports two evaluation methods: custom evaluation based on business outcomes (using your datasets and custom dimensions for evaluation) and baseline evaluation based on model general capabilities (using public standard datasets to quickly evaluate model basic capabilities).
Comparison dimension | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Applicable scenarios | Business scenario evaluation, custom standards | Model general capability validation (academic subjects, math, reasoning) | ||||||||||||||||
Evaluation standards | Custom evaluation dimensions (5 types) | Use preset evaluation rules | ||||||||||||||||
Need to create evaluation dimensions? | Required | Not required | ||||||||||||||||
Data source | See Custom evaluation - Evaluation data
| Preset standard datasets (provided by system), see Baseline evaluation - Evaluation data | ||||||||||||||||
Data format | Excel (2-3 columns: Prompt, Completion, Output) | No preparation needed—select directly | ||||||||||||||||
Flexibility | High—customize everything | Low—use industry standards | ||||||||||||||||
Evaluation workflow |
| |||||||||||||||||
- First-time use / Business scenario evaluation: Custom evaluation (more flexible).
- Validate general capabilities after fine-tuning: Baseline evaluation (quick comparison).
- Best practice: Use both together.
Custom evaluation
Supported models
Supports the following preset models (including fine-tuned versions):
- Qwen-QwQ/Max/Plus/Turbo/Coder/Math
- Qwen open-source versions (Qwen3, Qwen2.5, Qwen2, Qwen1.5)
- Tongyi Farui
- Third-party text generation models (abab6.5g, abab6.5t, etc.)
The list above is incomplete and subject to change. Always refer to the Create Evaluation Task page for the current list of Evaluate Model. Models shown in gray without an Apply button are currently not supported for evaluation.
Evaluation dimensions
Evaluation dimensions define scoring criteria—how you score models. You must create evaluation dimensions before custom evaluation.
-
Evaluation dimension types (five types)
Type
Scoring Method
Applicable Scenarios
Cost
Recommendation
Model Evaluation - Numeric
AI judge scores (1–5 points)
QA quality, content generation quality
Medium
⭐⭐⭐⭐⭐
Rule Evaluation - Text Similarity
Algorithm calculates similarity
Translation, summarization, rewriting
Low
⭐⭐⭐⭐
Model Evaluation - Classification
AI judge classifies (Pass/Fail)
Content moderation, correctness verification
Medium
⭐⭐⭐
Rule Evaluation - String Matching
Exact string matching
Function Calling, NL2SQL
Very low
⭐⭐⭐
Manual Evaluation - Classification
Manual labeling
Creativity, professional judgment
High
⭐⭐
- How to quickly select evaluation dimension types? One evaluation task can combine multiple evaluation dimensions.
Evaluation data
-
Data source types
Data type
Description
Workflow
Applicable scenarios
Evaluation dataset
Contains questions (prompts) and reference answers (completions)
The system calls the model for real-time inference, generates output, and then the scorer assigns a score.
Initial evaluation
Compare multiple models
The model has not yet generated results
Inference result set
Already contains model output
The system directly reads the output and the scorer assigns a score (no inference required).
The model has already generated results
Reduce inference costs
Baseline model for control groups
- Data format requirements: Excel
-
Evaluation dataset example:
Prompt
Completion
What is cloud computing?
Cloud computing is an on-demand pool of shared computing resources.
How does Python define functions?
You can use the def keyword to define functions.
-
Inference result set example:
Add one more Output column.
Prompt
Completion
Output
What is cloud computing?
Cloud computing is an on-demand pool of shared computing resources.
Cloud computing is a service model that delivers computing resources over the internet...
-
Data volume recommendations:
Phase
Recommended Quantity
Description
Small-scale validation
50–100 items
Verify configuration correctness.
Formal evaluation
200–500 items
Obtain reliable statistical results.
Comprehensive assessment
500+ items
Cover various edge cases.
Leaderboard
What is a leaderboard:
- Compare multiple models' performance on the same evaluation dimension.
- Visually display model ranking.
- Support horizontal comparison across multiple evaluation tasks.
- Compare multiple models' performance (e.g., GPT-4 vs Claude vs Qwen).
- Visually demonstrate model strengths/weaknesses in business scenarios.
- Share model comparison results within your team.
Core variables
Applicable scope: This section applies only to Model Evaluation - Numeric and Model Evaluation - Classification evaluation dimensions.
During large model evaluation, the judge model receives these three variables:
Variable name | Description | Example | Usage scenario |
|---|---|---|---|
| Original input: the user's question or instruction | What is cloud computing? | Required in all scenarios |
| Model output: the evaluated model's response | Cloud computing is a service delivered over... | Required in all scenarios |
| Reference answer: the standard answer or expected output | Cloud computing is an on-demand... | Use when standard answers exist |
Baseline evaluation
Supported models
Baseline evaluation only supportsfine-tuned models, not preset models (like Qwen-Max).Why only fine-tuned models:
- Baseline evaluation uses industry-standard datasets (like C-Eval, MMLU) to assess model capabilities.
- It primarily validates performance improvements after fine-tuning.
- It enables comparison with other industry models.
- Go to the Create Evaluation Task page.
- Select "Baseline Evaluation".
- Check the "Select Model" dropdown list.
- If the list is empty, you don't have fine-tuned models yet—you need to complete model fine-tuning and deployment first (see Model Deployment).
Evaluation data
Baseline evaluation requires no preparation of evaluation data—simply select system-provided data: standard leaderboards for Academic Subjects, Mathematics, or Reasoning (evaluation sets, logic, and scripts align with open-source leaderboard data and are regularly updated).
Academic Subjects: Evaluate model mastery of everyday common sense and academic knowledge.
Mathematics: Evaluate model ability to solve basic math problems.
Reasoning: Evaluate model Chinese natural language reasoning ability, such as determining logical relationships between sentences (entailment, contradiction).
Category | Evaluation Data | Description |
|---|---|---|
Academic Subjects |
| C-Eval primarily evaluates a model's understanding and application of Chinese text and includes more than 13,000 test questions across 52 disciplines, such as mathematics, physics, chemistry, history, geography, and literature. |
| MMLU primarily evaluates model knowledge across a broad range of domains, with over 57,000 test questions spanning 57 disciplines, including STEM, humanities, and social sciences. | |
| ARC primarily evaluates a model's ability to perform deep, complex reasoning using multidisciplinary knowledge, with over 7,900 test questions from biology, physics, chemistry, and other disciplines. | |
Mathematics |
| GSM8K primarily evaluates a model's ability to solve basic math problems and contains over 8,000 elementary school math problems. |
Reasoning |
| HellaSwag primarily evaluates a model's ability in everyday commonsense reasoning and contains over 70,000 reasoning test questions. |
| BBH (Big-Bench Hard) primarily evaluates a model’s ability to perform deep, complex reasoning and multi-step reasoning. It features 23 challenging test questions selected from BIG-Bench, covering logical reasoning, language understanding, and creative thinking. |
Baseline evaluation-specific report content
Baseline evaluation reports differ from custom evaluation reports by providing these unique elements:
-
Subject-specific score details (C-Eval and MMLU)
View model scores per subject, for example:
- Computer Science: 85%
- Law: 72%
- Medicine: 68%
- Mathematics: 90%
-
Capability radar chart
Visualize model capabilities across dimensions:
- Knowledge mastery
- Reasoning ability
- Mathematical ability
- Common-sense understanding
-
Industry comparison
Compare your model against industry standard models:
- GPT-4: XX points
- Your model: XX points
- Claude: XX points
System Prompt vs. Scorer Prompt differences
In model evaluation, two easily confused Prompt configurations serve different purposes:
Prompt type | Configuration location | Target | Action Phase | Required? | Cost attribution |
|---|---|---|---|---|---|
System Prompt | When creating Evaluation Task | Evaluated model | When the model generates answers | No | |
Scorer Prompt | When creating Model Evaluation - Numeric and Model Evaluation - Classification evaluation dimensions | Judge model | When the judge model scores answers | Yes |
Detailed explanation
-
System Prompt
- Purpose: Define role or behavior guidelines for the evaluated model.
-
Example scenarios:
- Evaluating customer service: "You are a professional customer service representative. Answer customer questions politely and accurately."
- Evaluating code assistant: "You are a programming expert. Provide accurate, concise code examples."
- When to use: Only fill when you need to assign a specific identity or constraints to the evaluated model—leave blank in most cases.
-
Scorer Prompt
- Purpose: Tell the judge model how to evaluate answer quality and what scoring criteria to use.
- Applicable scope: Required only for Model Evaluation - Numeric and Model Evaluation - Classification evaluation dimensions.
-
Configuration methods:
- Select system preset templates (Comprehensive Evaluation, Semantic Similarity, Sentiment Analysis, etc.).
- Write custom prompts (for specific business scenarios).
- Scorer Prompts can use ${prompt}, ${output}, ${completion} variables.
Vivid analogy
- System Prompt: Tells the "student" (evaluated model) what role to assume when taking the exam.
- Scorer Prompt: Tells the "grader" (judge model) how to grade the exam and what scoring standards to apply.
Scenario-based practice cases
Custom Evaluation:Model Evaluation - Numericevaluation dimension
Applicable scenarios: Situations requiring granular scoring, such as QA quality, content generation quality, or customer service dialogue quality.
Recommendations:
- Make scoring criteria specific—provide clear descriptions for each score level.
- Regularly spot-check high-scoring and low-scoring samples to verify accuracy.
- Avoid having all samples clustered in one score range.
- 1. Create evaluation dimension
- 2. Prepare test data
- 3. Create evaluation task
- 4. Check task status
Item | Content |
|---|---|
Dimension Name | Common-sense QA accuracy |
Description | Evaluate model accuracy in answering common-sense questions |
Type | Select Model Evaluation - Numeric |
Judge Model | Select Qwen-Max |
Scorer Template | Select Semantic Similarity |
Scoring Range | Keep default 0~5 |
Pass Threshold | 3.0 |
Template | Description | Applicable scenarios | Need custom Prompt? |
|---|---|---|---|
Comprehensive Evaluation (default) | A system preset template that scores across five dimensions: relevance, cultural sensitivity, information richness, clarity, and user engagement. | General conversations and customer service scenarios | No—use directly or modify as needed |
Semantic Similarity | A system preset template that evaluates the semantic closeness between the output and the reference answer. | QA systems and knowledge retrieval | No—use directly or modify as needed |
Custom Evaluation | Write your own prompt. | Specific business scenarios | Yes—write according to your requirements |
Custom evaluation: Large Model Evaluation - Classification evaluation dimension
Applicable scenarios: Situations requiring binary judgments like "yes/no" or "pass/fail", such as content moderation or answer correctness verification.
Recommendations:
- Clearly specify judgment conditions for each label in your Prompt.
- Explicitly require the judge model to output only label names.
- Regularly spot-check samples to ensure accuracy.
- 1. Create evaluation dimension
- 2. Prepare test data
- 3. Create evaluation task
- 4. Check task status
Item | Content |
|---|---|
Dimension Name | Sentiment analysis |
Description | Detect customer sentiment |
Type | Select Auto |
Judge Model | Select Qwen-Max |
Scorer Template | Select Sentiment Analysis |
Scoring Range | No need to modify—use default content |
Template | Description | Applicable scenarios | Need custom prompt? |
|---|---|---|---|
Standard Match (default) | A system preset template that checks whether the model output matches the reference answer. | Classification labels, function names, and fixed answers | No—use directly or modify as needed. |
Sentiment Analysis | A system preset template that analyzes text sentiment (positive, neutral, or negative). | User reviews, customer service dialogues, and public opinion monitoring | No—use directly or modify as needed. |
Custom Scorer | Write your own prompt. | Custom scenarios such as style judgment, compliance checks, and logic verification | Required. Write as needed. |
Custom evaluation: Large Model Evaluation - Numerical evaluation dimension
Applicable scenarios: Situations requiring multi-dimensional comprehensive scoring, such as answer quality assessment, content safety scoring, or translation quality scoring.
Recommendations:
- Set appropriate score ranges and pass thresholds—test small-scale first before batch evaluation.
- Choose suitable scorer templates: Comprehensive Evaluation for general scenarios, Semantic Similarity for scenarios with reference answers.
- When writing custom scorer Prompts, clearly define scoring criteria and value ranges.
- 1. Create evaluation dimension
- 2. Prepare test data
- 3. Create evaluation task
- 4. Check task status
Item | Content |
|---|---|
Dimension Name | Comprehensive quality assessment |
Description | Evaluate comprehensive quality of model responses |
Type | Select Model Evaluation - Numeric |
Judge Model | Select Qwen-Max |
Scorer Template | Select Comprehensive Evaluation (default) |
Prompt | No need to modify—use default content |
Scoring Range | Use default values 0-5 |
Pass Threshold | Use default value 3.0—samples scoring ≥3.0 are marked Pass |
Template | Description | Applicable scenarios | Need custom Prompt? |
|---|---|---|---|
Comprehensive Evaluation (default) | A system preset template that scores responses across five dimensions: relevance, cultural sensitivity, information richness, clarity, and user engagement. | General answer quality assessment | No—use directly or modify as needed |
Semantic Similarity | A system preset template that evaluates the semantic similarity between the model output and a reference answer. | Scenarios with clear reference answers, such as knowledge-based QA or translation | No—use directly or modify as needed |
Custom Evaluation | Write your own prompt—specify scoring criteria and value ranges in your prompt. | Custom scenarios, such as professional domain assessment or scoring based on specific metrics | Required. Write as needed. |
- Score range: Set minimum and maximum integer values for judge model scoring (default 0–5). The judge model will score each evaluation item within this range.
- Pass threshold: Scores ≥ threshold are marked Pass, scores below are marked Fail. Threshold range is constrained by score range and supports decimals (step size 0.1), default is 3.0.
Custom evaluation: Rule-based Evaluation - String Match evaluation dimension
Applicable scenarios: Fixed answer formats requiring exact matching, such as Function Calling or NL2SQL.
Recommendations:
- Normalize preprocessing: standardize case, trim whitespace.
- For synonyms, consider using text similarity scorers instead.
- Test matching rules with small samples first to verify expectations.
- 1. Create evaluation dimension
- 2. Prepare test data
- 3. Create evaluation task
- 4. Check task status
Item | Content |
|---|---|
Dimension Name | Function Calling validation |
Description | Check if model correctly invoked specified function |
Type | Select Rule Evaluation - String Matching |
Compare Text | Three-column row: left text box enter Enter model output variable in left text box, reference answer variable in right text box |
Rule | Behavior Description | Typical Application |
|---|---|---|
Equal | Exactly identical (case-insensitive) | City name validation, fixed answers |
Not Equal | Completely different (case-insensitive) | Prohibited word screening |
Contains | Contains the specified string | Keyword checking, essential information validation |
Starts With | Starts with the specified string | Prefix validation, format checking |
Ends With | Ends with the specified string | Suffix validation, terminator checking |
query_weather, it's marked Pass.Custom evaluation: Rule-based Evaluation - Text Similarity evaluation dimension
Applicable scenarios: Situations where answers vary in wording but share the same meaning, such as translation, summarization, or rewriting.
Recommendations:
- Test with small samples first—observe pass rate distribution before adjusting thresholds.
- Focus on word order and exact matching → choose BLEU.
- Focus on semantic understanding → choose Cosine.
- Need error tolerance → choose Fuzzy Match.
- Focus on key information coverage → choose ROUGE-L.
- 1. Create evaluation dimension
- 2. Prepare test data
- 3. Create evaluation task
- 4. Check task status
Item | Content |
|---|---|
Dimension Name | Translation quality |
Description | Evaluate Chinese-to-English translation quality |
Type | Select Rule Evaluation - Text Similarity |
Similarity algorithm | Select |
Scorer Template | Left: Enter model output variable in left text box, reference answer variable in right text box |
Pass Threshold | Set to 0.4—samples with similarity ≥0.4 are marked Pass (range 0~1) |
Algorithm | Characteristics | Best scenarios | Threshold recommendation |
|---|---|---|---|
ROUGE-L | Based on the longest common subsequence | Text summarization | 0.4–0.6 |
BLEU | Based on n-gram precision matching and sensitive to word order | Machine translation | 0.3–0.5 |
Cosine | Measures vector space similarity and captures semantics | Open QA and semantic understanding | 0.6–0.8 |
Fuzzy Match | Based on edit distance and offers high error tolerance | OCR and speech recognition | 0.7–0.9 |
Accuracy | Requires an exact match | Precise answer verification | 1.0 |
Custom evaluation: Manual Evaluation - Classification evaluation dimension
Applicable scenarios: Situations requiring human subjective judgment, such as creativity assessment or professional quality review.
Recommendations:
- Use manual evaluation when model output quality can't be precisely judged by algorithms or judge models alone.
- Document clear judgment criteria for each label in an external Evaluation Guide and train annotators for consistent understanding.
- Design labels to cover all possible output scenarios using concise, clear classification terms.
- Have multiple evaluators cross-annotate some samples to verify consistency.
- Avoid annotating too much data at once—process in batches to reduce fatigue effects.
- 1. Create evaluation dimension
- 2. Prepare test data
- 3. Create evaluation task
- 4. Check task status
Item | Content |
|---|---|
Dimension Name | Content creativity |
Description | Evaluates content creativity. |
Type | Select Manual Evaluation - Classification. |
Pass | Create two labels: "Excellent" and "Good." |
Fail | Create two labels: "Average" and "Poor." |
Compare multiple models using leaderboards
Applicable scenarios: Compare multiple models' performance on the same evaluation dimension and visually see model rankings through leaderboards.
Recommendations:
- Compare different specifications within the same model series (e.g., qwen-max, qwen-plus, qwen3.5-flash).
- Compare models from different vendors (e.g., Qwen vs DeepSeek).
- Compare different versions or configurations of the same model.
- 1. Create leaderboard
- 2. Add evaluation tasks to leaderboard
- 3. View leaderboard results
Item | Content |
|---|---|
Leaderboard Name | Common-sense QA accuracy leaderboard |
Evaluation Dimension | Select an existing evaluation dimension, such as Common-sense QA accuracy. |
Associate Optional Tasks | You can select existing evaluation tasks now or add them later. |
- Each leaderboard supports up to 50 evaluation tasks.
- After creating a leaderboard, you can't modify its evaluation dimension.
- All tasks in the same leaderboard use the same evaluation dimension to ensure fair comparison.
Baseline evaluation
Applicable scenarios: Quickly validate model general capabilities (academic knowledge, mathematical reasoning, scientific common sense) using industry-standard datasets.
Procedure:
- 1. Create evaluation task
- 2. Check task status
Item | Content |
|---|---|
Dimension Name | Baseline evaluation |
Evaluation Object | Select Qwen-Max. |
Type | Select the model to evaluate. |
Judge Model | Select system preset evaluation data (see Evaluation data). |
Advanced techniques
Best practices for evaluation dimension design
Single responsibility:
- Each evaluation dimension should focus on one evaluation goal.
- Avoid mixing multiple standards.
- Example: Don't assess both "accuracy" and "fluency" in the same dimension.
- Use objective, quantifiable standards whenever possible.
- Reduce subjectivity.
- Provide clear criteria for each score level.
- Continuously optimize Prompts based on evaluation feedback.
- Regularly spot-check to verify evaluation accuracy.
- Compare human and AI scoring to adjust evaluation standards.
Prompt writing techniques
See Prompt Best Practices and Plugins for Alibaba Cloud Bailian to learn how to maximize the model's performance without model fine-tuning.
Cost optimization recommendations
Choose evaluation methods wisely:
- Rule-based evaluation has the lowest cost—ideal for formatted output validation.
- Large model evaluation costs more—use for scenarios requiring semantic understanding.
- Manual evaluation is most expensive—reserve for cases requiring human judgment.
- For frequently evaluated baseline models, save outputs as inference result sets.
- Avoid repeated model calls to reduce inference costs.
- First validate configurations with small datasets (50–100 items).
- After confirming correct setup, scale to 200–500 items.
- Evaluate multiple models simultaneously to improve efficiency.
- Select multiple models in the same evaluation task.
Evaluation result analysis methods
Monitor score distribution:
- Check sample counts at each score level.
- Identify problematic clustering (e.g., all scores at 3 points).
- Adjust scoring standards or thresholds.
- Review high-scoring samples: understand model strengths.
- Review low-scoring samples: identify model weaknesses.
- Review borderline samples: refine evaluation standards.
- Don't just look at total scores—examine performance across different scenarios.
- Identify each model's strengths and weaknesses.
- Select appropriate models based on business priorities.
- Establish regular evaluation cycles.
- Record evaluation results after each fine-tuning.
- Analyze whether fine-tuning directions are correct.
Troubleshooting common issues
Evaluation results don't meet expectations:
- Verify test data represents your business scenarios.
- Review if evaluation dimension Prompts are clear.
- Try adjusting score ranges and thresholds.
- Consider changing judge models or scorer templates.
- Scoring criteria too broad → refine descriptions for each score level.
- Test data lacks diversity → add edge and abnormal samples.
- Judge model too conservative → switch to a more capable inference model.
- This is normal—models perform differently across dimensions.
- Weight dimensions according to business priorities.
- Don't rely on single dimensions—make decisions holistically.
- Model outputs completely unrelated to reference answers (model clearly lacks relevant background knowledge).
- Model outputs partially related to reference answers (model has background knowledge but it's outdated).
Billing information
Model evaluation costs come from two parts: evaluated model inference costs and judge model scoring costs.
Evaluated model inference costs
Model type | Billable? | Billing details |
|---|---|---|
Preset models (such as qwen-max, qwen-plus) | Yes | Billed based on token consumption |
Independently deployed models (fine-tuned and deployed) | No | No additional charges |
- System Prompt (if configured)
- Prompts from evaluation data (user questions)
- Model-generated responses (Output)
Judge model scoring costs (free for limited time)
Only large model evaluation (numerical, classification) involves judge model usage.
Complete cost calculation example
Scenario: Evaluate 100 data items using qwen-plus with Model Evaluation - Numeric, using qwen-max as judge model.
Assumptions:
- Average 50 tokens per Prompt.
- Average 200 tokens per model Output.
- Average 500 tokens per scorer Prompt (including scoring criteria).
- Average 5 tokens per judge model output.
-
Evaluated model (qwen-plus) cost:
- Input: 100 items × 50 tokens = 5,000 tokens
- Output: 100 items × 200 tokens = 20,000 tokens
- Assuming qwen-plus pricing: input ¥0.002/1K tokens, output ¥0.006/1K tokens
- Cost = 5,000/1000 × 0.002 + 20,000/1000 × 0.006 = ¥0.01 + ¥0.12 = ¥0.13
- Judge model (qwen-max) cost (free for limited time): ¥0
Pricing reference
Latest pricing: See Model Studio Console .
Note:
- Pricing may change—always refer to console display.
- Some models may have limited-time promotional offers.
Next steps
- If you're satisfied with the evaluated model's performance, start model invocation and integrate it into your business workflow.
- If you're dissatisfied with the evaluated model's performance, select another model for re-evaluation. You might also consider learning and obtaining the Alibaba Cloud Large Model ACA Certification. Its free accompanying courses help you systematically understand large model capabilities, application scenarios, and optimization techniques.