Continuously improve application quality through version comparison, debugging previews, user feedback, and high-quality evaluation data
Optimization supports multi-version comparison, prompt optimization based on debugging results and feedback, and prompt optimization based on high-quality evaluation data. This topic describes how to generate, review, and adopt optimized prompts based on debugging results and human feedback.
Before you start, make sure the application has been created and published successfully. Also prepare test questions that can expose problems with the application—for example, incomplete answers, an unexpected tone, failure to follow formatting requirements, or compliance risks.
Go to the Optimization page and click Select the Application to Optimize to start an optimization task.
In the Prompt Optimization area on the left, review the application's current prompt and confirm that the basics are complete—role definition, task objectives, response rules, boundary conditions, and output format. You can edit the prompt directly in the editor if changes are needed.
Click Comparison Mode to open the version-comparison debugging dialog. By default it shows one baseline group and one control group, and you can add up to two control groups. The initial configuration of the baseline and control groups matches the current draft configuration of the application. Click Settings in the top-right corner to view the baseline prompt or adjust the control-group prompts.
After making adjustments, enter a question and start a debugging task. The baseline and control groups each produce a response, so you can compare how different prompt versions behave. You can run multiple rounds of debugging, and modify the configuration at any time before continuing.
Once you have identified the best-performing version, click Adopt This Group's Configuration on that version. The system writes the group's prompt into the prompt editor.
Enter test questions in the Debugging Window on the right and send them to view the application's responses. Prioritize scenarios where problems commonly appear in real business use, for example:
Select one or more debugging results depending on the scope of the problem:
Click Optimize Prompt Based on Debugging Results to open the Optimize Prompt Based on Debugging Results dialog.
In Human Evaluation Feedback, describe the problems with the application's responses and the improvements you expect. Feedback should be specific—state "what is wrong" and "how it should be adjusted". Human feedback supplements business requirements that cannot be inferred directly from the debugging conversation. If you selected multiple debugging results, you can summarize their common problems and shared optimization goal in a single piece of feedback.
Click Start Optimization. The system analyzes the problem and generates a new prompt based on:
After confirming that the optimized prompt meets expectations, click Adopt. The system writes the result into the prompt editor.
When you have accumulated a batch of high-quality questions, reference answers, or historical responses, you can create an intelligent optimization task directly from an evaluation dataset.
On the Debugging & Optimization page, click Intelligent Optimization and select Optimize Prompt from Evaluation Dataset to open the Optimize from Evaluation Dataset dialog.
Map evaluation dataset fields to application fields:
After confirming the dataset version, data scope, and field mapping, click Start Optimization. The system creates an intelligent optimization task and you are taken to the task list.
Go to the Intelligent Optimization Tasks page to track task progress. When the task completes, click Intelligent Optimization Report to review the result. After confirming it meets expectations, click Adopt This Optimized Version, and the system writes the result into the prompt editor.
Once batch evaluation is complete, you can target low-scoring or failed samples from an evaluation task for optimization. Compared with using a dataset directly, this approach leverages existing evaluation results to identify common problems.
On the Debugging & Optimization page, click Intelligent Optimization and select Optimize Prompt from Evaluation Task to open the Optimize from Evaluation Task dialog.
Map evaluation task fields to application fields:
After confirming the evaluation task, data scope, and field mapping, click Start Optimization. The system creates an intelligent optimization task and you are taken to the task list.
Go to the Intelligent Optimization Tasks page to track task progress. When the task completes, click Intelligent Optimization Report to review the result. After confirming it meets expectations, click Adopt This Optimized Version, and the system writes the result into the prompt editor.
After adopting an optimization result, run debugging again and cover the following scenarios:
Before You Begin
Before you start, make sure the application has been created and published successfully. Also prepare test questions that can expose problems with the application—for example, incomplete answers, an unexpected tone, failure to follow formatting requirements, or compliance risks.
Open Optimization
Go to the Optimization page and click Select the Application to Optimize to start an optimization task.
Multi-Version Comparison
Review and Refine the Current Prompt
In the Prompt Optimization area on the left, review the application's current prompt and confirm that the basics are complete—role definition, task objectives, response rules, boundary conditions, and output format. You can edit the prompt directly in the editor if changes are needed.
Start a Version Comparison
Click Comparison Mode to open the version-comparison debugging dialog. By default it shows one baseline group and one control group, and you can add up to two control groups. The initial configuration of the baseline and control groups matches the current draft configuration of the application. Click Settings in the top-right corner to view the baseline prompt or adjust the control-group prompts.
Debug Across Versions
After making adjustments, enter a question and start a debugging task. The baseline and control groups each produce a response, so you can compare how different prompt versions behave. You can run multiple rounds of debugging, and modify the configuration at any time before continuing.
Adopt a Configuration
Once you have identified the best-performing version, click Adopt This Group's Configuration on that version. The system writes the group's prompt into the prompt editor.
Adopting only updates the prompt currently being edited—it does not publish the application automatically. After adopting, you can continue editing manually or run another comparison.
Debugging and Feedback Optimization
Start a Debugging Conversation
Enter test questions in the Debugging Window on the right and send them to view the application's responses. Prioritize scenarios where problems commonly appear in real business use, for example:
- The response omits rules, conditions, or exceptions.
- The tone is stiff and does not match the application's persona.
- The output does not follow the required format, or is verbose and poorly structured.
- The response ignores knowledge base content and is not grounded.
- Safety, compliance, or business-boundary constraints are missing.
Select the Debugging Results to Optimize
Select one or more debugging results depending on the scope of the problem:
- One result: Best for a clearly defined problem that a single case can illustrate. The optimization dialog shows "Current debugging result".
- Multiple results: Best for problems that recur or require an overall judgment. After entering batch-selection mode, select representative results; the optimization dialog shows "Selected debugging results (N)".
Provide Human Feedback
Click Optimize Prompt Based on Debugging Results to open the Optimize Prompt Based on Debugging Results dialog.
In Human Evaluation Feedback, describe the problems with the application's responses and the improvements you expect. Feedback should be specific—state "what is wrong" and "how it should be adjusted". Human feedback supplements business requirements that cannot be inferred directly from the debugging conversation. If you selected multiple debugging results, you can summarize their common problems and shared optimization goal in a single piece of feedback.
Generate and Review the Optimization Result
Click Start Optimization. The system analyzes the problem and generates a new prompt based on:
- The application's current prompt.
- The current or selected debugging conversations.
- The human evaluation feedback.
Adopt the Optimization Result
After confirming that the optimized prompt meets expectations, click Adopt. The system writes the result into the prompt editor.
Adopting only updates the prompt currently being edited—it does not publish the application automatically. After adopting, you can continue editing manually or select other debugging results and optimize again.
Optimize from an Evaluation Dataset
When you have accumulated a batch of high-quality questions, reference answers, or historical responses, you can create an intelligent optimization task directly from an evaluation dataset.
Create an Optimization Task
On the Debugging & Optimization page, click Intelligent Optimization and select Optimize Prompt from Evaluation Dataset to open the Optimize from Evaluation Dataset dialog.
Select the Dataset and Records
- Select the evaluation dataset and version to use for optimization. If needed, click View Details to confirm the field schema, data content, and version are correct.
- In the data list, select the records to include in this optimization. The list supports paginated selection and shows the number of selected records. If you made a wrong selection, deselect records or choose a different dataset.
Configure Field Mapping
Map evaluation dataset fields to application fields:
| Application field | Required | Description |
|---|---|---|
| Application input | Yes | Maps to the user question, request, or other application input field. |
| Application output | No | Maps to the application's actual response, the reference answer, or the field used to judge the target output. |
| Trajectory | No | Configure when the dataset contains call-chain, step, or trajectory information. |
Start Optimization
After confirming the dataset version, data scope, and field mapping, click Start Optimization. The system creates an intelligent optimization task and you are taken to the task list.
Review the Result and Adopt in One Click
Go to the Intelligent Optimization Tasks page to track task progress. When the task completes, click Intelligent Optimization Report to review the result. After confirming it meets expectations, click Adopt This Optimized Version, and the system writes the result into the prompt editor.
Adopting only updates the prompt currently being edited—it does not publish the application automatically. After adopting, you can continue editing manually or select other debugging results and optimize again.
Optimize from an Evaluation Task
Once batch evaluation is complete, you can target low-scoring or failed samples from an evaluation task for optimization. Compared with using a dataset directly, this approach leverages existing evaluation results to identify common problems.
Create an Optimization Task
On the Debugging & Optimization page, click Intelligent Optimization and select Optimize Prompt from Evaluation Task to open the Optimize from Evaluation Task dialog.
Select the Task and Records
- Select an evaluation task whose status is Completed. If needed, click View Task Details to confirm that the task's application version, evaluation data, and grader configuration match your optimization goal.
- In the data list, select the records to include in this optimization. The list supports paginated selection and shows the number of selected records. If you made a wrong selection, deselect records or choose a different evaluation task.
Configure Field Mapping
Map evaluation task fields to application fields:
| Application field | Required | Description |
|---|---|---|
| Application input | Yes | Maps to the user question, request, or other application input field. |
| Application output | No | Maps to the application's actual response, the reference answer, or the field used to judge the target output. |
| Trajectory | No | Configure when the evaluation task contains call-chain, step, or trajectory information. |
Start Optimization
After confirming the evaluation task, data scope, and field mapping, click Start Optimization. The system creates an intelligent optimization task and you are taken to the task list.
Review the Result and Adopt in One Click
Go to the Intelligent Optimization Tasks page to track task progress. When the task completes, click Intelligent Optimization Report to review the result. After confirming it meets expectations, click Adopt This Optimized Version, and the system writes the result into the prompt editor.
Adopting only updates the prompt currently being edited—it does not publish the application automatically. After adopting, you can continue editing manually or select other debugging results and optimize again.
Validate and Publish
After adopting an optimization result, run debugging again and cover the following scenarios:
- The original problem scenario: Confirm the problem has been resolved.
- Normal scenarios: Confirm existing capabilities have not regressed.
- Boundary scenarios: Confirm abnormal input, insufficient information, or conflicting instructions are handled reasonably.
Optimization Tips
- Focus on one main problem at a time: Optimize content completeness, tone, formatting, and compliance in separate batches—it is easier to judge the effect of each change.
- Use real, representative conversations: Turn high-frequency production issues and failed evaluation samples into debugging questions, or use test questions that reliably reproduce the problem.
- Make feedback explicit and constrained: Rather than "answer better", state exactly what must be included, what must not appear, and the expected output form.
- Adopt first, then validate, then publish: Do not judge effectiveness from the optimized text alone—validate actual response quality through debugging.
- Keep humans in the loop: Intelligent optimization may not understand implicit business rules. Review carefully whenever knowledge standards, permissions, safety, or compliance requirements are involved.
FAQ
Does the application take effect immediately after I adopt an optimization result?
Does the application take effect immediately after I adopt an optimization result?
No. Adopting only writes the optimized content into the current prompt. You still need to validate it through debugging and click Publish before the new version takes effect.
Should I select one debugging result or several?
Should I select one debugging result or several?
Select one result for a single, clearly defined problem. If the same type of problem recurs across multiple questions, select several results to help the system identify the common pattern. Multiple results should focus on the same optimization goal.
How do I write high-quality human feedback?
How do I write high-quality human feedback?
Use the pattern "symptom + specific gap or error + expected behavior". For example: "The response only covers the general refund policy and omits the exception for home appliances; please list the general rules, applicable conditions, and special-category exceptions in full."
Do I need to debug again after optimizing?
Do I need to debug again after optimizing?
Yes. Prompt changes can affect other response scenarios. Validate the original problem, normal flows, and boundary inputs together, and confirm no new problems were introduced before publishing.