Manual evaluation dimensions use evaluators to manually classify model outputs by label. They suit scenarios that require subjective judgment, such as creative writing and professional domain judgment.
Manual evaluation overview
Manual evaluation uses evaluators to review model inference results one by one and annotate Pass/Fail labels. It does not involve a judge model, incurs no scoring costs, but requires labor input and takes longer. It suits scenarios that machines can hardly evaluate accurately:
- Creative writing, tone and style, and other subjective quality judgments.
- Compliance review of professional domain content.
Create manual evaluation dimensions
Log on to the Model Studio console, choose Model Evaluation in the left-side navigation pane, switch to the Evaluation Dimensions tab, and click Create Evaluation Dimension, and in the dialog, click + Create Custom Evaluation Dimension to go to the custom configuration page. After you fill in the common fields (dimension name up to 20 characters, description up to 100 characters, and the type cannot be changed after creation), complete the configuration as follows:
- Configure the Pass label: define the meaning of the label for pass.
- Configure the Fail label: define the meaning of the label for fail. Pass and Fail labels must not duplicate each other.
- Click Save.