---
id: 22fdc181-fef2-4455-b387-bbc3750dda31
---

# Vault too many pending tokens incident on kubernetes
---

The Vault too many pending tokens incident refers to an issue where the Vault server has too many pending tokens. This can happen when the number of tokens generated exceeds the limit of what the server can handle, causing a backlog of requests. As a result, users may experience difficulty accessing certain resources or functions that require token authentication. This incident type typically requires the attention of a system administrator or engineer to investigate and resolve the underlying cause, such as increasing the token limit or optimizing the token generation process.

### Parameters
```shell
export NAMESPACE="PLACEHOLDER"

export VAULT_POD_NAME="PLACEHOLDER"

export VAULT_CONTAINER_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export CASSANDRA_POD_NAME="PLACEHOLDER"

export VAULT_ADDRESS="PLACEHOLDER"

export MAX_CONNECTIONS="PLACEHOLDER"
```

## Debug

### Check the status of the Vault pod
```shell
kubectl get pods -n ${NAMESPACE}
```

### Check logs of Vault container
```shell
kubectl logs ${VAULT_POD_NAME} -n ${NAMESPACE} ${VAULT_CONTAINER_NAME}
```

### Check the current resource usage of the Vault pod
```shell
kubectl top pods ${VAULT_POD_NAME} -n ${NAMESPACE}
```

### Check the current resource usage of the Cassandra pod
```shell
kubectl top pods ${CASSANDRA_POD_NAME} -n ${NAMESPACE}
```

### Check the current resource limits and requests for the Vault and Cassandra pods
```shell
kubectl describe pods ${VAULT_POD_NAME} -n ${NAMESPACE}

kubectl describe pods ${CASSANDRA_POD_NAME} -n ${NAMESPACE}
```

### High number of concurrent requests to Vault API that leads to the generation of too many pending tokens.
```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

POD_NAME=${VAULT_POD_NAME}

CONTAINER_NAME=${VAULT_CONTAINER_NAME}

VAULT_ADDR=${VAULT_ADDRESS}

MAX_CONNECTIONS=${MAX_CONNECTIONS}



# Get the number of current connections

CURRENT_CONNECTIONS=$(kubectl exec $POD_NAME -n $NAMESPACE -c $CONTAINER_NAME -- sh -c "curl -s $VAULT_ADDR/v1/sys/stats | grep 'vault' | awk '{print $2}'")



# Check if the current connections exceeds the maximum allowed connections

if [ $CURRENT_CONNECTIONS -gt $MAX_CONNECTIONS ]

then

  echo "Too many concurrent connections to Vault API. Current connections: $CURRENT_CONNECTIONS"

else

  echo "Current connections to Vault API: $CURRENT_CONNECTIONS"

fi


```

## Repair

### Increase the token creation rate limit on Vault to reduce the number of pending tokens.
```shell


#!/bin/bash



# Set the namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT_NAME}



# Get the current token creation rate limit

RATE_LIMIT=$(kubectl get deployment -n $NAMESPACE $DEPLOYMENT -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="VAULT_TOKENS_RATE_LIMIT")].value}')



# Increase the rate limit by 10 tokens per second

NEW_RATE_LIMIT=$((RATE_LIMIT + 10))



# Update the deployment with the new rate limit

kubectl set env deployment -n $NAMESPACE $DEPLOYMENT VAULT_TOKENS_RATE_LIMIT=$NEW_RATE_LIMIT


```

The Vault too many pending tokens incident refers to an issue where the Vault server has too many pending tokens. This can happen when the number of tokens generated exceeds the limit of what the server can handle, causing a backlog of requests. As a result, users may experience difficulty accessing certain resources or functions that require token authentication. This incident type typically requires the attention of a system administrator or engineer to investigate and resolve the underlying cause, such as increasing the token limit or optimizing the token generation process.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

The Vault Sealed incident type occurs when an instance of the Vault has been sealed due to an error or issue. This can cause disruption to the system and requires immediate attention to resolve the issue and unseal the Vault instance. It is important to identify the root cause of the issue in order to prevent it from happening again in the future.


Vault Sealed Incident

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

This incident type involves issues with persistent volume claims in a Kubernetes cluster. This could be due to storage problems or other configuration issues. It is important to resolve these issues promptly to ensure that the Kubernetes cluster is functioning properly.


Kubernetes - Troubleshooting Failed Persistent Volume Claims

```shell
export NAMESPACE="PLACEHOLDER"

export VAULT_POD_NAME="PLACEHOLDER"

export VAULT_CONTAINER_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export CASSANDRA_POD_NAME="PLACEHOLDER"

export VAULT_ADDRESS="PLACEHOLDER"

export MAX_CONNECTIONS="PLACEHOLDER"
```


### Check the status of the Vault pod

```shell
kubectl get pods -n ${NAMESPACE}
```

### Check logs of Vault container

```shell
kubectl logs ${VAULT_POD_NAME} -n ${NAMESPACE} ${VAULT_CONTAINER_NAME}
```

### Check the current resource usage of the Vault pod

```shell
kubectl top pods ${VAULT_POD_NAME} -n ${NAMESPACE}
```

### Check the current resource usage of the Cassandra pod

```shell
kubectl top pods ${CASSANDRA_POD_NAME} -n ${NAMESPACE}
```

### Check the current resource limits and requests for the Vault and Cassandra pods

```shell
kubectl describe pods ${VAULT_POD_NAME} -n ${NAMESPACE}

kubectl describe pods ${CASSANDRA_POD_NAME} -n ${NAMESPACE}
```

### High number of concurrent requests to Vault API that leads to the generation of too many pending tokens.

```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

POD_NAME=${VAULT_POD_NAME}

CONTAINER_NAME=${VAULT_CONTAINER_NAME}

VAULT_ADDR=${VAULT_ADDRESS}

MAX_CONNECTIONS=${MAX_CONNECTIONS}



# Get the number of current connections

CURRENT_CONNECTIONS=$(kubectl exec $POD_NAME -n $NAMESPACE -c $CONTAINER_NAME -- sh -c "curl -s $VAULT_ADDR/v1/sys/stats | grep 'vault' | awk '{print $2}'")



# Check if the current connections exceeds the maximum allowed connections

if [ $CURRENT_CONNECTIONS -gt $MAX_CONNECTIONS ]

then

  echo "Too many concurrent connections to Vault API. Current connections: $CURRENT_CONNECTIONS"

else

  echo "Current connections to Vault API: $CURRENT_CONNECTIONS"

fi


```


### Increase the token creation rate limit on Vault to reduce the number of pending tokens.

```shell


#!/bin/bash



# Set the namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT_NAME}



# Get the current token creation rate limit

RATE_LIMIT=$(kubectl get deployment -n $NAMESPACE $DEPLOYMENT -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="VAULT_TOKENS_RATE_LIMIT")].value}')



# Increase the rate limit by 10 tokens per second

NEW_RATE_LIMIT=$((RATE_LIMIT + 10))



# Update the deployment with the new rate limit

kubectl set env deployment -n $NAMESPACE $DEPLOYMENT VAULT_TOKENS_RATE_LIMIT=$NEW_RATE_LIMIT


```


Vault too many pending tokens incident on kubernetes

Overview

Parameters

Debug

Check the status of the Vault pod

Check logs of Vault container

Check the current resource usage of the Vault pod

Check the current resource usage of the Cassandra pod

Check the current resource limits and requests for the Vault and Cassandra pods

High number of concurrent requests to Vault API that leads to the generation of too many pending tokens.

Repair

Increase the token creation rate limit on Vault to reduce the number of pending tokens.

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

Vault Sealed Incident

Kubernetes pods not starting - Deployment issue

Kubernetes Pods Pending

Support