---
id: 14ab065c-eb92-4d87-8ec2-00d342065fee
---

# High container volume usage incident for ECS.
---

This incident type refers to a situation where the usage of container volume exceeds the threshold limit, which is typically set at 80%. This can lead to performance issues and potentially cause downtime for the service or application running in the container. It requires immediate attention from the assigned engineer to investigate and resolve the issue.

### Parameters
```shell
export CLUSTER_NAME="PLACEHOLDER"

export LOG_GROUP_NAME="PLACEHOLDER"

export LOG_STREAM_NAME="PLACEHOLDER"

export EBS_VOLUME_ID="PLACEHOLDER"

export INSTANCE_ID="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"
```

## Debug

### 1. Get the list of instances running in the container.
```shell
aws ecs list-container-instances --cluster ${CLUSTER_NAME}
```

### 2. Get the list of tasks running in the container.
```shell
aws ecs list-tasks --cluster ${CLUSTER_NAME} --container-instance ${INSTANCE_ARN}
```

### 3. Get the container details from the task.
```shell
aws ecs describe-tasks --cluster ${CLUSTER_NAME} --tasks ${TASK_ARN}
```

### 4. Get the container logs.
```shell
aws logs get-log-events --log-group-name ${LOG_GROUP_NAME} --log-stream-name ${LOG_STREAM_NAME} --start-time ${START_TIME} --end-time ${END_TIME}
```

### A sudden spike in user traffic causing an unexpected increase in the container's volume usage.
```shell


#!/bin/bash



# Set AWS region and instance ID

region=${REGION}

instance_id=${INSTANCE_ID}



# Get the container ID

container_id=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --region $region --container-name ${CONTAINER_NAME} --query 'taskArns[0]' --output text)



# Get the current volume usage

volume_usage=$(aws ecs describe-tasks --cluster ${CLUSTER_NAME} --region $region --tasks $container_id --query 'tasks[0].containers[0].filesystemUsage' --output text)



# Get the maximum volume capacity

volume_capacity=$(aws ecs describe-tasks --cluster ${CLUSTER_NAME} --region $region --tasks $container_id --query 'tasks[0].containers[0].filesystems[0].size' --output text)



# Calculate the percentage of volume usage

volume_percentage=$(( volume_usage * 100 / volume_capacity ))



# Check if the volume usage is above the threshold limit

if [ $volume_percentage -gt 80 ]; then

  echo "The container's volume usage is higher than the threshold limit."

  # Add logic to handle the incident here, such as scaling up the container or investigating the root cause

else

  echo "The container's volume usage is normal."

fi


```

## Repair

### Increase the storage capacity of the container volume to accommodate the growing data.
```shell


#!/bin/bash



# Set AWS region and EBS volume ID

region="${AWS_REGION}"

volume_id="${EBS_VOLUME_ID}"



# Get the current size of the EBS volume

current_size=$(aws ec2 describe-volumes --region $region --volume-ids $volume_id | jq -r '.Volumes[0].Size')



# Calculate the new size (increase by 10 GB)

new_size=$((current_size + 10))



# Modify the EBS volume size

aws ec2 modify-volume --region $region --volume-id $volume_id --size $new_size


```


This incident type refers to a situation where the usage of container volume exceeds the threshold limit, which is typically set at 80%. This can lead to performance issues and potentially cause downtime for the service or application running in the container. It requires immediate attention from the assigned engineer to investigate and resolve the issue.


​
AWS EBS (Elastic Block Store) Burst Credits Exhausted is an incident that occurs when an Amazon EBS volume has exhausted its burst credit balance. This happens when the volume has been used continuously at its maximum IOPS (input/output operations per second) rate for an extended period of time, causing the volume to consume its burst credits faster than it can earn them. As a result, the volume's performance is significantly degraded until the burst credits are replenished.
​


AWS EBS Burst Credits Exhausted

The Vault too many pending tokens incident refers to an issue where the Vault server has too many pending tokens. This can happen when the number of tokens generated exceeds the limit of what the server can handle, causing a backlog of requests. As a result, users may experience difficulty accessing certain resources or functions that require token authentication. This incident type typically requires the attention of a system administrator or engineer to investigate and resolve the underlying cause, such as increasing the token limit or optimizing the token generation process.


Vault too many pending tokens incident on kubernetes

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

A high CPU load incident occurs when the CPU usage of a host or server exceeds a certain threshold, typically 80%. This can cause slowdowns, unresponsiveness, or even crashes in the affected system. It is critical to address this issue promptly to minimize impact on users and prevent further damage to the system.


High CPU Load Incident

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

```shell
export CLUSTER_NAME="PLACEHOLDER"

export LOG_GROUP_NAME="PLACEHOLDER"

export LOG_STREAM_NAME="PLACEHOLDER"

export EBS_VOLUME_ID="PLACEHOLDER"

export INSTANCE_ID="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"
```


### 1. Get the list of instances running in the container.

```shell
aws ecs list-container-instances --cluster ${CLUSTER_NAME}
```

### 2. Get the list of tasks running in the container.

```shell
aws ecs list-tasks --cluster ${CLUSTER_NAME} --container-instance ${INSTANCE_ARN}
```

### 3. Get the container details from the task.

```shell
aws ecs describe-tasks --cluster ${CLUSTER_NAME} --tasks ${TASK_ARN}
```

### 4. Get the container logs.

```shell
aws logs get-log-events --log-group-name ${LOG_GROUP_NAME} --log-stream-name ${LOG_STREAM_NAME} --start-time ${START_TIME} --end-time ${END_TIME}
```

### A sudden spike in user traffic causing an unexpected increase in the container's volume usage.

```shell


#!/bin/bash



# Set AWS region and instance ID

region=${REGION}

instance_id=${INSTANCE_ID}



# Get the container ID

container_id=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --region $region --container-name ${CONTAINER_NAME} --query 'taskArns[0]' --output text)



# Get the current volume usage

volume_usage=$(aws ecs describe-tasks --cluster ${CLUSTER_NAME} --region $region --tasks $container_id --query 'tasks[0].containers[0].filesystemUsage' --output text)



# Get the maximum volume capacity

volume_capacity=$(aws ecs describe-tasks --cluster ${CLUSTER_NAME} --region $region --tasks $container_id --query 'tasks[0].containers[0].filesystems[0].size' --output text)



# Calculate the percentage of volume usage

volume_percentage=$(( volume_usage * 100 / volume_capacity ))



# Check if the volume usage is above the threshold limit

if [ $volume_percentage -gt 80 ]; then

  echo "The container's volume usage is higher than the threshold limit."

  # Add logic to handle the incident here, such as scaling up the container or investigating the root cause

else

  echo "The container's volume usage is normal."

fi


```


### Increase the storage capacity of the container volume to accommodate the growing data.

```shell


#!/bin/bash



# Set AWS region and EBS volume ID

region="${AWS_REGION}"

volume_id="${EBS_VOLUME_ID}"



# Get the current size of the EBS volume

current_size=$(aws ec2 describe-volumes --region $region --volume-ids $volume_id | jq -r '.Volumes[0].Size')



# Calculate the new size (increase by 10 GB)

new_size=$((current_size + 10))



# Modify the EBS volume size

aws ec2 modify-volume --region $region --volume-id $volume_id --size $new_size


```


High container volume usage incident for ECS.

Overview

Parameters

Debug

1. Get the list of instances running in the container.

2. Get the list of tasks running in the container.

3. Get the container details from the task.

4. Get the container logs.

A sudden spike in user traffic causing an unexpected increase in the container's volume usage.

Repair

Increase the storage capacity of the container volume to accommodate the growing data.

Learn more

Related Runbooks

AWS EBS Burst Credits Exhausted

Vault too many pending tokens incident on kubernetes

Tomcat High JDBC Connection Pool Usage Incident.

High CPU Load Incident

Support