Skip to main content

API key leak investigation

How to investigate a suspected API key leak: locate and disable the leaked key, identify anomalous periods from usage and billing, confirm call behavior with audit and inference logs, review management operations in ActionTrail, then complete damage assessment, evidence collection, and rotation.

Investigation approach

If the leak is confirmed, go to the API key management page first and reset or delete the leaked key to stop the loss, then follow the steps on this page to assess the impact; if the leak is uncertain, start with the usage check in step 2. A leak investigation answers five questions, each mapped to a console information source:

Question to answer

Where to look

What you get

Which key leaked? Who created it? What is its current status?

API key management page

Key ID, masked key, description, owning account, creation time; reset, edit, delete, and cost viewing actions

How much was this key called, and how much did it cost?

Model usage, cost overview, and bills

Call counts, token usage, daily usage curves, and amounts grouped by API key

Where did the calls come from, which models were called, and did they succeed?

Audit logs

Request ID, time, model, token usage, latency, status code, and error code; filterable by API key

What was the input and output of the calls?

Inference logs (must be enabled in advance)

Complete prompts and responses (up to 128K)

Which management operations has the key gone through?

ActionTrail

Operation event records at the console and OpenAPI level

Proceed in the following order: 
Suspected leak
  ├─ Step 1: locate the key and immediately reset or delete it (containment)
  ├─ Step 2: identify anomalous periods from usage and cost curves
  ├─ Step 3: confirm call sources and behavior in audit logs (input and output available if inference logs were enabled)
  ├─ Step 4: review management operations such as key creation and deletion
  └─ Step 5: summarize the damage, rotate credentials, and fix the leak source

Step 1: Locate and disable the leaked key

Go to the Alibaba Cloud Model Studio API Key page and select a region in the upper-right corner: keys are independent per region and cannot be used across regions. When the leak is confirmed, prioritize resetting or deleting the key to stop the loss immediately; editing to narrow permissions only fits situations where the leak is uncertain and you need an observation window. Locate the key in the API key list and use the actions in its row:
  • Reset: issues a new value for the key, and the old value stops working immediately.
  • Edit: narrows permissions (IP whitelist, model scope); see API key leak prevention best practices.
  • Delete: takes effect immediately and is irreversible.
  • View costs: jumps to the cost and usage view of the key.
This page shows the key identity information: key ID, masked API key (for example sk-ws-H.YHXL****PgonKozeC4RM-Nm), description, owning account, and creation time. The description and owning account are the first clues to which application or owner the key belongs to; the complete plaintext is usually not viewable: new keys are displayed in full only once at creation, while some legacy keys can still be copied directly from the list, depending on what the list actually offers. For identification, rely on the masked value, ID, and description.
Deleting a key does not cancel associated asynchronous tasks: tasks that have not finished will continue to run and incur charges, and must be terminated separately. See Asynchronous task billing when overdue. During the investigation, you can also enable "stop immediately when the budget is reached" for the key as a second containment measure; see Budget management.
Two common pitfalls when handling the incident:
  1. Creating a new key does not automatically invalidate the old key; the old key must be handled separately.
  2. Do not put the new key back into an environment that has not finished being investigated, or you are handing over the new credential again.

Step 2: Check usage and costs to identify anomalous periods

The most direct signal of a leak is calls or costs that should not exist. On the Model Studio console, choose Cost and Usage > Model Usage, set the grouping dimension to API Key, and compare the usage of each key to spot the anomalous one at a glance:
  • Time granularity: minute (up to 1 day), hour (up to 7 days), day (up to 31 days), and month (up to 12 months); use hourly granularity to locate sudden high-cost calling.
  • Data delay: call counts are delayed by less than 1 minute, token usage and costs by about 10 minutes; recent abuse may appear later, so do not rule out suspicion just because no data shows yet.
  • Detail rows offer View costs for the associated amounts; for cost overview and bill accounting, see View cost overview and Query bills.
For metric definitions and operations, see View model usage. Rule out normal business fluctuation first: compare the anomalous time, models, and call volume with releases, batch jobs, stress tests, and scheduled tasks of the business, and check whether the deployment region, egress IPs, or third-party tools using the key changed. Overseas IPs, new IPs, and usage spikes are signals that require verification, not conclusive evidence of a leak. Match the response to the level of suspicion: for a mere alert or mild anomaly, check the call sources in audit logs first; once the usage curve confirms calls from unfamiliar sources, reset or delete the key immediately and continue evidence collection. Taking the leak confirmation time as the endpoint, look back 7 days (hourly) and 31 days (daily) to compare the key's historical baseline; the identified anomalous window defines the scope of the subsequent log investigation.

Step 3: Check call logs to confirm call behavior

Model Studio call logs fall into two categories with different capability boundaries; distinguish them during investigation.

Audit logs

Audit logs are enabled by default and stored on the platform side with no configuration required, making them the primary investigation tool. View them on the model monitoring page:
  • Recorded fields: Request ID, time, model, token usage, latency, status, and error code, but not input or output content.
  • Filtering and search: filter by time (up to 30 days), model, API key ID, status, and inference type, or search by Request ID.
  • Usage: confirm whether the key's calls within the leak window match expected behavior, such as large volumes of 4xx/5xx, abnormal calling rhythms, or calls to unfamiliar models.
For details, see Logs.

Inference logs

Inference logs contain complete prompts and responses plus intermediate steps (up to 128K). They must be enabled in advance under data delivery and delivered to your own log service (SLS; enabling involves cloud product authorization and log service costs), and only apply to future data: historical calls before enabling cannot be traced back, and logs generated while delivery is off are not backfilled. For enabling and regional availability, see Enable inference logs. If inference logs were not enabled, earlier calls can only be investigated through the field-level information in audit logs.

Tracing boundaries

The platform can help verify anomalous behavior based on call logs, but the calling IP, model, and time information is usually insufficient to directly confirm the actual user or the original leak channel; further tracing requires evidence from your code repositories, servers, terminals, and third-party services. The deliverable of an investigation is an evidence chain and impact assessment, not an attribution conclusion; for illegal activities, preserve evidence in accordance with the law and handle it through appropriate channels.

Step 4: Review management operation history

Lifecycle operations on API keys such as creation and deletion are aggregated in Alibaba Cloud ActionTrail. Go to the ActionTrail event query page and filter by service (Model Studio) and time range:
  • Look up the key's creation events to confirm who created the key and when, verifying ownership.
  • Look up the key's deletion events to confirm whether anyone deleted keys around the leak window, for example an attacker or insider attempting to erase traces.
Confirm the key status and operation history through ActionTrail first, then investigate specific behavior with the call logs.

Step 5: Damage assessment, evidence collection, and recovery

  1. Summarize the damage: combine the anomalous window (usage curves), call sources and statuses (audit logs), and amounts (bills) into an impact conclusion.
  2. Define the impact scope: assess by the key's authorization scope. Holding a Model Studio API key is not equivalent to holding the account password or full cloud resource permissions; the actual impact is limited to the authorized calling scope of the key. Note that a Model Studio API key and an Alibaba Cloud AccessKey (AK) are different credentials: if an AK leaked, follow the RAM AK handling process instead.
  3. Preserve masked evidence: when saving screenshots and Request ID lists, mask key values and account IDs, and stop spreading the complete key in any channel.
  4. Review asynchronous tasks: deleting a key does not cancel submitted asynchronous tasks; terminate cancellable tasks and track the costs of non-cancellable ones.
  5. Determine the rotation scope: invalidate and replace confirmed leaked keys as soon as possible; for the remaining keys, judge by exposure scope, and rotate together the keys stored in the same exposed environment.
  6. Recovery order: fix the leak source with the high-risk locations checklist, then create a new key in a trusted environment with reconfigured permissions and budget, update credential references in services, jobs, and third-party integrations, confirm the new key works, and finally verify the old key is invalid.
Before abnormal costs are fully investigated, do not simply raise the budget or restore calling after turning off loss mitigation; complete the investigation above first to confirm the leak source is fixed.

FAQ

Detection and notification involve time gaps, and notification time alone cannot tell whether the anomaly is still ongoing; historical events may also only be discovered and notified during later reviews. Receiving a notification does not mean the old key can still call successfully: if the old key is handled and recent usage is normal, verify the call time and key identifier in the notification and ignore it.
Repetition alone does not confirm a leak. Check the call time, key identifier, and anomaly details of each alert to distinguish new calls, repeated notifications of the same event, and historical records.
No. A new key is displayed in full only once at creation and can no longer be viewed or copied afterwards (some legacy keys can still be copied directly from the list, depending on what the list actually offers).
Inference logs only apply to future data; historical data cannot be traced back. Earlier calls can only be investigated through the field-level information in audit logs.
Token usage and costs are delayed by about 10 minutes; the usage page also only counts pay-as-you-go token-based calls, excluding training and pre-provisioned throughput (PTU). Failure counts and throttling errors are available on the model monitoring page.
Both return 429. A budget-exhausted error reads "The budget configured in Budget Management has been exhausted..."; see Error codes. A throttling 429 is triggered by rate or quota limits; distinguish them with the throttling error count metric in model monitoring.
For your own analysis, deliver logs to your own SLS log store and search or export them there; see Data delivery.
They are different credentials: a Model Studio API key only accesses the model services within its authorization scope; an Alibaba Cloud AccessKey authenticates cloud API calls with permissions determined by the account and authorization configuration. Confirm the credential type first, then go to the corresponding product's management entry; do not mix up the processes.