Prevent High Azure Insights Costs: Configuration Tips

What Went Wrong

Recently, I encountered a costly issue with my application. A misconfigured Polly retry policy caused over 80 million retries to a 3rd-party API within just two days. Each retry failure generated logs in the AppEvents and AppExceptions tables of my Azure Log Analytics workspace linked to Application Insights. This caused a big increase in data ingestion, leading to an unexpected rise in my Azure costs. Normally, I pay about 70 EUR per month, but this month I had to pay 150 EUR, effectively doubling my costs.

The main problem was that the Polly retry logic failed to handle the rate limit headers correctly because the API, for some reason, sent a RetryAfter value of 0 instead of providing an appropriate wait time. This led to requests being retried too frequently, despite the API's advice to pause, which in turn flooded the logging system with telemetry data.

Steps Taken to Prevent This in the Future

To address this issue and avoid a similar situation in the future, I implemented several measures, including daily ingestion caps, telemetry sampling, alerts, and data purging. Here’s how you can configure these features in Application Insights to ensure cost-effective monitoring and logging.

1. Configure Daily Caps in Log Analytics

Daily caps are an effective way to prevent runaway data ingestion costs. If your application unexpectedly generates excessive telemetry, the daily cap will stop data ingestion once the cap is reached, protecting you from unexpected billing spikes.

How to Set a Daily Cap:

Go to your Log Analytics Workspace in the Azure portal.
Navigate to Usage and estimated costs in the left menu.
Select the Daily cap tab.
Set the desired cap in GB (e.g., 2 GB/day).
Save the settings.

Azure will stop data ingestion for the day once the cap is reached. Note that ingestion resumes the next day.

2. Implement Telemetry Sampling in Application Insights

Telemetry sampling reduces the volume of data sent to Application Insights by collecting a representative subset of telemetry data. This ensures you retain insights into application behavior while keeping ingestion volumes under control.

How to Enable Telemetry Sampling:

Open your Application Insights resource in the Azure portal.
Navigate to Sampling under the Configure section.
Enable Adaptive Sampling:
- Adaptive Sampling dynamically adjusts the sampling rate based on the volume of incoming telemetry.
Adjust sampling rates for specific telemetry types (e.g., reduce detailed logs while keeping critical exceptions).

This ensures that only essential logs are sent to the workspace, reducing data ingestion without losing valuable insights.

3. Configure Alerts to Monitor Unexpected Behavior

Alerts help you detect spikes in telemetry data before they escalate into high costs. By monitoring key metrics, such as data ingestion rates or exception counts, you can act quickly to resolve the underlying issue.

How to Set Up Alerts:

In your Application Insights resource, go to Alerts in the left menu.
Click + New alert rule.
Configure the alert:
- Scope: Select your Application Insights resource.
- Condition: Choose a signal like Exceptions > Count or Data Ingestion > Volume.
- Threshold: Set a threshold (e.g., trigger when exceptions exceed 1000 in 5 minutes).
- Action Group: Define an email or SMS notification for the alert.
Save and enable the alert.

This allows you to receive notifications when unusual patterns emerge, such as a sudden increase in exceptions or retries.

4. Purge Unnecessary Data with the REST API

After identifying the source of the excessive logs, I needed to remove the unnecessary data to clean up my workspace, hoping it would lower my bill. However, that's not the case. You are billed for the data you logged at the end of the day, so purging the data after two days didn't affect my invoice. Still, you might want to do this, so here's how I did it step-by-step:

Step 1: Assign the Data Purger Role

Go to your Log Analytics Workspace in the Azure portal.
Navigate to Access Control (IAM).
Add a role assignment:
- Role: Data Purger
- Assign access to: Your Application Insights or service principal.

Step 2: Generate an Access Token

Use Azure CLI or Postman to generate a token:
```
 az account get-access-token --resource https://management.azure.com/
```
Use the token in the Authorization header of your requests.

Step 3: Send a Purge Request

Use the following POST endpoint:

 POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/purge?api-version=2023-09-01

Include the following request body (set the value to whatever you need to purge):

 {
   "table": "AppExceptions",
   "filters": [
     {
       "column": "Type",
       "operator": "=",
       "value": "Polly.RateLimit.RateLimitRejectedException"
     }
   ]
 }

Monitor the operation with the returned operationId. In my case it took more than 24 hours for the purge to execute.

Conclusion: Why Configuring Application Insights is Essential

This experience highlighted the importance of properly configuring Application Insights to prevent high costs and ensure effective telemetry management. By implementing daily caps, telemetry sampling, alerts, and data purging, you can:

Prevent runaway data ingestion costs.
Reduce unnecessary logs while retaining critical insights.
Detect and respond to unusual application behavior early.
Maintain a clean and manageable telemetry dataset.

By taking these steps proactively, you can safeguard your monitoring budget and ensure your application remains performant and cost-effective.

Configuring Azure Application Insights to Prevent High Costs