---
id: 60bf1d4b-8a6d-47d8-89a2-8a0c879222d3
---

# Vault cluster health incident on kubernetes
---

The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.

### Parameters
```shell
export DEPLOYMENT_NAME="PLACEHOLDER"

export VAULT_POD_NAME="PLACEHOLDER"

export LOG_FILE_PATH="PLACEHOLDER"

export SERVICE_URL="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"
```

## Debug

### Get the list of pods running in the Kubernetes cluster
```shell
kubectl get pods
```

### Get the logs of a specific pod
```shell
kubectl logs ${VAULT_POD_NAME}
```

### Check the status of a Kubernetes deployment
```shell
kubectl rollout status deployment/${DEPLOYMENT_NAME}
```

### Describe a Kubernetes pod to get more information about its state
```shell
kubectl describe pod ${VAULT_POD_NAME}
```

### Check if the Vault cluster is running and accessible
```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} bash -- vault status
```

### Check the logs of the Vault server to see if there are any errors
```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} bash -- vault server -log-level=debug
```

### Check the health of the Vault cluster nodes
```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} bash -- vault operator raft list-peers
```

### A network issue that is affecting the connectivity or performance of the Vault cluster instance.
```shell


#!/bin/bash



# Define the namespace, pod name, and container name for the Vault instance

NAMESPACE=${NAMESPACE}

POD_NAME=${VAULT_POD_NAME}

CONTAINER_NAME=${CONTAINER_NAME}



# Check the logs of the Vault container

kubectl logs $POD_NAME -c $CONTAINER_NAME -n $NAMESPACE



# Check the network connectivity between the Vault container and other services

kubectl exec $POD_NAME -c $CONTAINER_NAME -n $NAMESPACE -- ping ${SERVICE_NAME}



# Check the network performance between the Vault container and other services

kubectl exec $POD_NAME -c $CONTAINER_NAME -n $NAMESPACE -- curl --connect-timeout 5 --max-time 10 ${SERVICE_URL}


```

## Repair

### Check the logs of the Vault cluster instance to identify the root cause of the issue. This could involve looking for error messages or other indicators of problems with the instance.
```shell


#!/bin/bash



# Set variables

VAULT_POD=${VAULT_POD_NAME}

LOG_FILE=${LOG_FILE_PATH}



# Get logs for the Vault pod

kubectl logs $VAULT_POD > $LOG_FILE



# Search for error messages

if grep -q "error" $LOG_FILE; then

    echo "Error messages found in logs"

    # Perform additional remediation steps as needed

else

    echo "No error messages found in logs"

fi


```

### Restart the Vault cluster instance to see if this resolves the issue. This could involve stopping and starting the instance, or rebooting the server if necessary.
```shell


#!/bin/bash



# Set the namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT_NAME}



# Use rolling restart to restart the pods

kubectl rollout restart deployment/${DEPLOYMENT} -n ${NAMESPACE}


```

The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


The Vault too many pending tokens incident refers to an issue where the Vault server has too many pending tokens. This can happen when the number of tokens generated exceeds the limit of what the server can handle, causing a backlog of requests. As a result, users may experience difficulty accessing certain resources or functions that require token authentication. This incident type typically requires the attention of a system administrator or engineer to investigate and resolve the underlying cause, such as increasing the token limit or optimizing the token generation process.


Vault too many pending tokens incident on kubernetes

The Vault Sealed incident type occurs when an instance of the Vault has been sealed due to an error or issue. This can cause disruption to the system and requires immediate attention to resolve the issue and unseal the Vault instance. It is important to identify the root cause of the issue in order to prevent it from happening again in the future.


Vault Sealed Incident

This incident type is characterized by the detection of unauthorized access to the Kubernetes API server. This unauthorized access potentially enables attackers to manipulate cluster resources. Such incidents can lead to serious consequences, as unauthorized access allows attackers to make changes to the Kubernetes cluster, which may compromise the security and integrity of the entire system. Therefore, it is crucial to promptly detect and address such incidents to ensure the security and proper functioning of the Kubernetes cluster.


Unauthorized Access to Kubernetes API Server Detected

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

This incident type occurs when the Kubernetes node status is not OK. It means that the scheduler cannot place pods on the node due to some underlying issue with the node's health. This incident can impact the availability and performance of the applications running on the Kubernetes cluster. Immediate attention is required to resolve this incident to restore the normal functioning of the Kubernetes cluster.


Kubernetes Node Status Not OK

```shell
export DEPLOYMENT_NAME="PLACEHOLDER"

export VAULT_POD_NAME="PLACEHOLDER"

export LOG_FILE_PATH="PLACEHOLDER"

export SERVICE_URL="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"
```


### Get the list of pods running in the Kubernetes cluster

```shell
kubectl get pods
```

### Get the logs of a specific pod

```shell
kubectl logs ${VAULT_POD_NAME}
```

### Check the status of a Kubernetes deployment

```shell
kubectl rollout status deployment/${DEPLOYMENT_NAME}
```

### Describe a Kubernetes pod to get more information about its state

```shell
kubectl describe pod ${VAULT_POD_NAME}
```

### Check if the Vault cluster is running and accessible

```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} bash -- vault status
```

### Check the logs of the Vault server to see if there are any errors

```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} bash -- vault server -log-level=debug
```

### Check the health of the Vault cluster nodes

```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} bash -- vault operator raft list-peers
```

### A network issue that is affecting the connectivity or performance of the Vault cluster instance.

```shell


#!/bin/bash



# Define the namespace, pod name, and container name for the Vault instance

NAMESPACE=${NAMESPACE}

POD_NAME=${VAULT_POD_NAME}

CONTAINER_NAME=${CONTAINER_NAME}



# Check the logs of the Vault container

kubectl logs $POD_NAME -c $CONTAINER_NAME -n $NAMESPACE



# Check the network connectivity between the Vault container and other services

kubectl exec $POD_NAME -c $CONTAINER_NAME -n $NAMESPACE -- ping ${SERVICE_NAME}



# Check the network performance between the Vault container and other services

kubectl exec $POD_NAME -c $CONTAINER_NAME -n $NAMESPACE -- curl --connect-timeout 5 --max-time 10 ${SERVICE_URL}


```


### Check the logs of the Vault cluster instance to identify the root cause of the issue. This could involve looking for error messages or other indicators of problems with the instance.

```shell


#!/bin/bash



# Set variables

VAULT_POD=${VAULT_POD_NAME}

LOG_FILE=${LOG_FILE_PATH}



# Get logs for the Vault pod

kubectl logs $VAULT_POD > $LOG_FILE



# Search for error messages

if grep -q "error" $LOG_FILE; then

    echo "Error messages found in logs"

    # Perform additional remediation steps as needed

else

    echo "No error messages found in logs"

fi


```

### Restart the Vault cluster instance to see if this resolves the issue. This could involve stopping and starting the instance, or rebooting the server if necessary.

```shell


#!/bin/bash



# Set the namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT_NAME}



# Use rolling restart to restart the pods

kubectl rollout restart deployment/${DEPLOYMENT} -n ${NAMESPACE}


```


Vault cluster health incident on kubernetes

Overview

Parameters

Debug

Get the list of pods running in the Kubernetes cluster

Get the logs of a specific pod

Check the status of a Kubernetes deployment

Describe a Kubernetes pod to get more information about its state

Check if the Vault cluster is running and accessible

Check the logs of the Vault server to see if there are any errors

Check the health of the Vault cluster nodes

A network issue that is affecting the connectivity or performance of the Vault cluster instance.

Repair

Check the logs of the Vault cluster instance to identify the root cause of the issue. This could involve looking for error messages or other indicators of problems with the instance.

Restart the Vault cluster instance to see if this resolves the issue. This could involve stopping and starting the instance, or rebooting the server if necessary.

Learn more

Related Runbooks

Vault too many pending tokens incident on kubernetes

Vault Sealed Incident

Unauthorized Access to Kubernetes API Server Detected

Kubernetes Statefulset Replicas Monitoring Incident

Support