---
id: c92e398d-0164-4d03-a370-ab84959282ab
---

# Check GPU Statistics
---

GPUs are frequently used to speed up highly parallel processes such as rendering or machine learning applications. This runbook contains commands that check the compute, memory utilization, and other metrics for your GPUs and helps to debug potential issues related to this.

### Parameters
```shell
export NVIDIA_VERSION="PLACEHOLDER"
```

## Debug

### Check the configuration of the graphics card
```shell
sudo lshw -C display
```

### List all NVIDIA GPUs
```shell
nvidia-smi --list-gpus
```

### Displays NVIDIA gpu info, including driver version, number of GPUs, product name, display mode, etc.
```shell
nvidia-smi -q
```

### Display gpu utilization.
```shell
nvidia-smi -q -i 0 -d UTILIZATION -l 1
```

### Display gpu performance.
```shell
nvidia-smi -q -d PERFORMANCE
```

### Display gpu clock information.
```shell
nvidia-smi -q -d CLOCK
```

### Display gpu power information.
```shell
nvidia-smi -q -d POWER
```

### Display gpu memory usage.
```shell
nvidia-smi -q -d MEMORY
```

### Display and constantly refresh all info about the NVIDIA GPU
```shell
watch -n 1 nvidia-smi
```

## Repair

### Update or install NVIDIA driver.
```shell
sudo apt-get install nvidia-driver-${NVIDIA_VERSION}
```


GPUs are frequently used to speed up highly parallel processes such as rendering or machine learning applications. This runbook contains commands that check the compute, memory utilization, and other metrics for your GPUs and helps to debug potential issues related to this.


This incident type refers to an unexpected termination of the Spark driver program during the runtime of a job. The driver program is responsible for coordinating the execution of a Spark job and if it crashes, the entire job is affected. This can result in data loss and downtime, and requires investigation and troubleshooting to identify the root cause of the issue.


Spark driver program crash during job runtime.

This incident type refers to a situation where a Spark cluster experiences performance bottlenecks when it is subjected to peak loads. In other words, the Spark cluster struggles to handle the high volume of requests it receives during times of heavy traffic or increased demand. This can lead to slower processing times, delays, or even system crashes. Identifying and resolving the root cause of the bottlenecks is crucial to ensure the smooth functioning of the Spark cluster during peak loads.


Spark cluster bottlenecks during peak loads.

This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

Data skew slowing down a Spark job is a common incident that occurs when there is an uneven distribution of data across partitions. This can lead to some partitions handling a much larger amount of data than others, causing delays and slowing down the entire job. The result is a significant decrease in performance and a longer processing time for the Spark job.


Data Skew Slowing Down Spark Job.

This incident type involves the detection of high CPU usage on a MongoDB host, often triggered by an alert monitor. The incident may be indicated by anomalous CPU usage values that are more than two standard deviations above predicted values during a specific time period, such as the last 15 minutes. The incident can cause performance issues and may require investigation to identify the root cause and resolve the problem.


High CPU usage on MongoDB host

```shell
export NVIDIA_VERSION="PLACEHOLDER"
```


### Check the configuration of the graphics card

```shell
sudo lshw -C display
```

### List all NVIDIA GPUs

```shell
nvidia-smi --list-gpus
```

### Displays NVIDIA gpu info, including driver version, number of GPUs, product name, display mode, etc.

```shell
nvidia-smi -q
```

### Display gpu utilization.

```shell
nvidia-smi -q -i 0 -d UTILIZATION -l 1
```

### Display gpu performance.

```shell
nvidia-smi -q -d PERFORMANCE
```

### Display gpu clock information.

```shell
nvidia-smi -q -d CLOCK
```

### Display gpu power information.

```shell
nvidia-smi -q -d POWER
```

### Display gpu memory usage.

```shell
nvidia-smi -q -d MEMORY
```

### Display and constantly refresh all info about the NVIDIA GPU

```shell
watch -n 1 nvidia-smi
```


### Update or install NVIDIA driver.

```shell
sudo apt-get install nvidia-driver-${NVIDIA_VERSION}
```


Check GPU Statistics

Overview

Parameters

Debug

Check the configuration of the graphics card

List all NVIDIA GPUs

Displays NVIDIA gpu info, including driver version, number of GPUs, product name, display mode, etc.

Display gpu utilization.

Display gpu performance.

Display gpu clock information.

Display gpu power information.

Display gpu memory usage.

Display and constantly refresh all info about the NVIDIA GPU

Repair

Update or install NVIDIA driver.

Learn more

Related Runbooks

Spark driver program crash during job runtime.

Spark cluster bottlenecks during peak loads.

High Latency Incident for Spark Job Execution.

Data Skew Slowing Down Spark Job.

Support