---
id: 9ba048eb-339f-47c0-8200-ddf8993a612f
---

# Vault Sealed Incident
---

The Vault Sealed incident type occurs when an instance of the Vault has been sealed due to an error or issue. This can cause disruption to the system and requires immediate attention to resolve the issue and unseal the Vault instance. It is important to identify the root cause of the issue in order to prevent it from happening again in the future.

### Parameters
```shell
export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export VAULT_POD_NAME="PLACEHOLDER"

export VAULT_CONTAINER_NAME="PLACEHOLDER"

export VAULT_NAME="PLACEHOLDER"
```

## Debug

### List all pods in the namespace
```shell
kubectl get pods -n ${NAMESPACE}
```

### Check if the pod is running
```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Check the logs of the pod
```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE}
```

### Check the status of the Vault server
```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault status
```

### Check if the Vault server is sealed
```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault status | grep Sealed
```

### Check the Vault server logs
```shell
kubectl logs ${VAULT_POD_NAME} -n ${NAMESPACE}
```

### Check the Vault server configuration
```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault operator init -status
```

### Check the Vault server health
```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault operator check
```

### Check if the Vault server has sufficient resources
```shell
kubectl describe pod ${VAULT_POD_NAME} -n ${NAMESPACE} | grep -E "Limits|Requests"
```

### Check the Vault server's storage
```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault storage check
```

### Check the Vault server's audit logs
```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault audit list
```

### Issues with the Vault service configuration leading to a failure in the Vault instance.
```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

VAULT_POD_NAME=${VAULT_POD_NAME}

VAULT_CONTAINER_NAME=${VAULT_CONTAINER_NAME}



# Check the status of the Vault pod

VAULT_STATUS=$(kubectl get pods -n $NAMESPACE $VAULT_POD_NAME -o jsonpath='{.status.phase}')



if [ "$VAULT_STATUS" != "Running" ]; then

  echo "Vault pod is not running. Current status: $VAULT_STATUS"

  exit 1

fi



# Check the logs of the Vault container

VAULT_LOGS=$(kubectl logs -n $NAMESPACE $VAULT_POD_NAME -c $VAULT_CONTAINER_NAME)



if [[ $VAULT_LOGS == *"configuration file: permission denied"* ]]; then

  echo "Vault configuration file permission denied"

  exit 1

elif [[ $VAULT_LOGS == *"configuration file: no such file or directory"* ]]; then

  echo "Vault configuration file not found"

  exit 1

elif [[ $VAULT_LOGS == *"Error initializing storage of type"* ]]; then

  echo "Vault storage not properly configured"

  exit 1

elif [[ $VAULT_LOGS == *"Error initializing listener of type"* ]]; then

  echo "Vault listener not properly configured"

  exit 1

else

  echo "No issues found with Vault configuration"

  exit 0

fi


```

## Repair

### unseal the vault
```shell


#!/bin/bash



# Set the namespace and name of the Vault instance

NAMESPACE=${NAMESPACE}

VAULT_NAME=${VAULT_NAME}



# Get the pod name of the Vault instance

POD_NAME=$(kubectl get pods -n $NAMESPACE -l app=$VAULT_NAME -o jsonpath='{.items[0].metadata.name}')



# Unseal the Vault instance

kubectl exec -it $POD_NAME -n $NAMESPACE -- vault operator unseal


```

The Vault Sealed incident type occurs when an instance of the Vault has been sealed due to an error or issue. This can cause disruption to the system and requires immediate attention to resolve the issue and unseal the Vault instance. It is important to identify the root cause of the issue in order to prevent it from happening again in the future.


The Vault too many pending tokens incident refers to an issue where the Vault server has too many pending tokens. This can happen when the number of tokens generated exceeds the limit of what the server can handle, causing a backlog of requests. As a result, users may experience difficulty accessing certain resources or functions that require token authentication. This incident type typically requires the attention of a system administrator or engineer to investigate and resolve the underlying cause, such as increasing the token limit or optimizing the token generation process.


Vault too many pending tokens incident on kubernetes

The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

This incident type refers to a failure in one or more Spark executors during the execution of a job. Spark executors are worker processes that run computations and store data in memory or on disk. When an executor fails, it can cause the entire job to fail or result in degraded performance. This type of incident can occur for a variety of reasons, such as hardware or network issues, memory errors, or software bugs.


Spark executor failure during job execution.

This incident type refers to a situation where the SSL certificate probe failed for a particular instance, which means that the SSL certificate for that instance could not be verified or was found to be invalid. This could be a potential security risk and needs to be investigated and resolved promptly.


SSL certificate probe failure

This incident type refers to the failure of a systemd service on a particular host instance. The incident could be triggered by various causes such as a software bug, hardware failure, or system overload. This type of incident can cause downtime or service disruption to the affected host instance, which may require immediate resolution to restore normal operations.


Host systemd service crashed (instance) incident.

```shell
export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export VAULT_POD_NAME="PLACEHOLDER"

export VAULT_CONTAINER_NAME="PLACEHOLDER"

export VAULT_NAME="PLACEHOLDER"
```


### List all pods in the namespace

```shell
kubectl get pods -n ${NAMESPACE}
```

### Check if the pod is running

```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Check the logs of the pod

```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE}
```

### Check the status of the Vault server

```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault status
```

### Check if the Vault server is sealed

```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault status | grep Sealed
```

### Check the Vault server logs

```shell
kubectl logs ${VAULT_POD_NAME} -n ${NAMESPACE}
```

### Check the Vault server configuration

```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault operator init -status
```

### Check the Vault server health

```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault operator check
```

### Check if the Vault server has sufficient resources

```shell
kubectl describe pod ${VAULT_POD_NAME} -n ${NAMESPACE} | grep -E "Limits|Requests"
```

### Check the Vault server's storage

```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault storage check
```

### Check the Vault server's audit logs

```shell
kubectl exec -it ${VAULT_POD_NAME} -n ${NAMESPACE} -- vault audit list
```

### Issues with the Vault service configuration leading to a failure in the Vault instance.

```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

VAULT_POD_NAME=${VAULT_POD_NAME}

VAULT_CONTAINER_NAME=${VAULT_CONTAINER_NAME}



# Check the status of the Vault pod

VAULT_STATUS=$(kubectl get pods -n $NAMESPACE $VAULT_POD_NAME -o jsonpath='{.status.phase}')



if [ "$VAULT_STATUS" != "Running" ]; then

  echo "Vault pod is not running. Current status: $VAULT_STATUS"

  exit 1

fi



# Check the logs of the Vault container

VAULT_LOGS=$(kubectl logs -n $NAMESPACE $VAULT_POD_NAME -c $VAULT_CONTAINER_NAME)



if [[ $VAULT_LOGS == *"configuration file: permission denied"* ]]; then

  echo "Vault configuration file permission denied"

  exit 1

elif [[ $VAULT_LOGS == *"configuration file: no such file or directory"* ]]; then

  echo "Vault configuration file not found"

  exit 1

elif [[ $VAULT_LOGS == *"Error initializing storage of type"* ]]; then

  echo "Vault storage not properly configured"

  exit 1

elif [[ $VAULT_LOGS == *"Error initializing listener of type"* ]]; then

  echo "Vault listener not properly configured"

  exit 1

else

  echo "No issues found with Vault configuration"

  exit 0

fi


```


### unseal the vault

```shell


#!/bin/bash



# Set the namespace and name of the Vault instance

NAMESPACE=${NAMESPACE}

VAULT_NAME=${VAULT_NAME}



# Get the pod name of the Vault instance

POD_NAME=$(kubectl get pods -n $NAMESPACE -l app=$VAULT_NAME -o jsonpath='{.items[0].metadata.name}')



# Unseal the Vault instance

kubectl exec -it $POD_NAME -n $NAMESPACE -- vault operator unseal


```


Vault Sealed Incident

Overview

Parameters

Debug

List all pods in the namespace

Check if the pod is running

Check the logs of the pod

Check the status of the Vault server

Check if the Vault server is sealed

Check the Vault server logs

Check the Vault server configuration

Check the Vault server health

Check if the Vault server has sufficient resources

Check the Vault server's storage

Check the Vault server's audit logs

Issues with the Vault service configuration leading to a failure in the Vault instance.

Repair

unseal the vault

Learn more

Related Runbooks

Vault too many pending tokens incident on kubernetes

Vault cluster health incident on kubernetes

Spark executor failure during job execution.

SSL certificate probe failure

Support