Observability

The multi-node architecture inside Alibaba Cloud Bailian applications introduces several challenges for subsequent development, including:

• Tracing internal application call chains
• Viewing model response latency
• Inspecting model reasoning processes

Using the Application Observability feature, you can observe end-to-end processing flows (e.g., large model invocations, MCP invocations, and vector retrieval) for Alibaba Cloud Bailian applications in your business space—and view monitoring metrics such as latency and token counts, as well as rate-limiting metrics such as QPM (updated on a per-minute basis).

Application Observability currently does not provide an API.

​

Supported Applications

Agent applications (including Agent 1.0 and Agent 2.0) and workflow applications.

Application Observability currently does not support agent applications created via the Assistant API.

​

Getting Started

​

Prerequisites

When using Application Observability for the first time, click Application Observability Configuration in the upper-right corner of the Application Observability page and complete the following steps:

1. Authorize the OpenTelemetry service role permissions for Alibaba Cloud Tracing Analysis.
2. Activate the Alibaba Cloud Tracing Analysis OpenTelemetry service.
3. Initialize the LogStore for Alibaba Cloud Tracing Analysis OpenTelemetry storage.

​

How to Use

​

1. Enable Application Observability

Navigate to Application Observability, find the target application in the list, and click Enable Observability. If your previously created application does not appear in the list, it may be because:

• The application has not yet been published.
• The application does not belong to the current business space.

​

2. Start Observing

1. Once observability is enabled for an application, all prompts submitted to this application—and their corresponding data and metrics—are automatically traced and synchronized to Application Observability (on a per-minute basis).

   Clicking Disable Observability stops synchronization of trace data. Upon re-enabling, only newly generated data will be synchronized.

2. Click View Details under the Actions column to enter the application details page. By default, the Trace List tab is displayed, showing summary information—including prompt content, output, latency, invocation timestamp, and token count—for all prompts submitted to this application over the past period (up to 30 days). You can search by Request ID, Trace ID, or Span ID, and filter results by custom time range.
3. Click a specific node's Trace ID to view detailed information—including raw data and annotation records (if available). Child nodes (if any) can be expanded.

​

3. Export Data

In the application details page, click Export Data in the upper-right corner of the Trace List tab to export data matching your current filters—available in JSONL or Excel format.

​

4. View Monitoring Statistics

In the application details page, click the Monitoring Statistics tab to view and download performance monitoring data for the application, including:

• Overview: Invocation count, total tokens, average invocation latency, average time to first token, and error count—each with period-over-period comparison.
• Invocation Analysis: Trend charts for invocation count, error count, and average invocation latency.
• Token Analysis: Total tokens (all / input / output), average input tokens per request, average output tokens per request, average time to first token, average invocation latency, and per-model token consumption trends.
• Model Analysis: Per-model invocation distribution, invocation trends, error trends, and TTFT trends.
• Tool Invocation Analysis: Per-tool invocation distribution, invocation trends, latency trends, and error count trends.

You can customize the time range (up to 30 days) and aggregation granularity (per minute, per hour, or per day).

​

5. View Rate-Limiting Statistics

In the application details page, click the Rate-Limiting Statistics tab to view and download rate-limiting monitoring data for the application, including:

• Application: QPM throttling, QPM throttling counts, and more.
• Quick configuration of alert rules, and support for filing a ticket to request a rate-limit increase.

​

Add to Evaluation Set

Application Observability supports directly adding Span data to an evaluation set for subsequent application evaluation. This enables you to use real production traffic as evaluation samples—building evaluation sets that better reflect actual business scenarios.

1. In the Application Observability list, click View Details next to the target application to open the Span list page. Then click Batch Operations, and select the Span entries you want to add to the evaluation set (multiple selections supported).
2. Click Add to Evaluation Set, which opens the configuration page:
   • Target Evaluation Set and Target Evaluation Set Name: Choose an existing evaluation set or create a new one.
   • Import Mode: Choose between Append Data or Full Overwrite.
   • Field Mapping: Map fields from Span data to corresponding fields in the evaluation set.
3. After completing configuration, click Start Import.

Field mapping supports selecting from the full set of Span parameters. Each evaluation set supports up to 50 field mappings.