---
id: 1baeefd8-c41a-4611-a243-1c21e7cc6a2a
---
# High ITCP Connections on Kafka Cluster
---

When there are high ITCP connections on a Kafka cluster, it means that there are many connections that have been established but not yet closed. This can cause a number of issues, including decreased performance and potential instability of the cluster. It is important to identify and address these connections in a timely manner to prevent further issues.

### Parameters
```shell
export KAFKA_PORT="PLACEHOLDER"

export KAFKA_PID="PLACEHOLDER"

export PATH_TO_KAFKA_HOME="PLACEHOLDER"

export CURRENT_NODE_ID="PLACEHOLDER"

export NUMBER_OF_NEW_NODES_TO_ADD="PLACEHOLDER"

export CURRENT_NODE_HOSTNAME="PLACEHOLDER"
```

## Debug

### Check current number of established ITCP connections
```shell
netstat -anp | grep ${KAFKA_PORT} | grep -c ESTABLISHED
```

### Check the maximum number of allowed ITCP connections
```shell
sysctl net.ipv4.tcp_max_syn_backlog
```

### Check the current number of half-open ITCP connections
```shell
ss -s | grep "TCP:" | awk '{print "Half-open connections: " $4}'
```

### Check the number of sockets in TIME_WAIT state
```shell
netstat -an | grep ${KAFKA_PORT} | grep TIME_WAIT | wc -l
```

### Check the number of sockets in CLOSE_WAIT state
```shell
netstat -an | grep ${KAFKA_PORT} | grep CLOSE_WAIT | wc -l
```

### Check the current number of file descriptors used by Kafka processes
```shell
lsof | grep ${KAFKA_PID} | wc -l
```

### Check the current amount of memory used by Kafka processes
```shell
ps -p ${KAFKA_PID} -o rss=
```

### Check the current amount of CPU usage by Kafka processes
```shell
top -p ${KAFKA_PID}
```

## Repair

### Increase Kafka Cluster capacity: One of the possible reasons for high ITCP connections is that the Kafka cluster is reaching its capacity limit. In this case, we can consider increasing the capacity of the Kafka cluster to handle more connections.
```shell


#!/bin/bash



# Define variables

KAFKA_HOME=${PATH_TO_KAFKA_HOME}

NEW_NODE_COUNT=${NUMBER_OF_NEW_NODES_TO_ADD}



# Stop Kafka services

sudo systemctl stop kafka



# Add new Kafka nodes

for i in $(seq 1 $NEW_NODE_COUNT);

do

    # Create new Kafka node

    sudo mkdir -p $KAFKA_HOME/kafka-node-$i

    sudo chown -R kafka:kafka $KAFKA_HOME/kafka-node-$i



    # Copy Kafka configuration files

    sudo cp -r $KAFKA_HOME/config $KAFKA_HOME/kafka-node-$i/

    sudo cp -r $KAFKA_HOME/logs $KAFKA_HOME/kafka-node-$i/



    # Update Kafka configuration files

    sudo sed -i "s/broker.id=${CURRENT_NODE_ID}/broker.id=$i/" $KAFKA_HOME/kafka-node-$i/config/server.properties

    sudo sed -i "s/#listeners/listeners/" $KAFKA_HOME/kafka-node-$i/config/server.properties

    sudo sed -i "s/#advertised.listeners/advertised.listeners/" $KAFKA_HOME/kafka-node-$i/config/server.properties

    sudo sed -i "s/${CURRENT_NODE_HOSTNAME}/$HOSTNAME/" $KAFKA_HOME/kafka-node-$i/config/server.properties

done



# Start Kafka services

sudo systemctl start kafka


```



When there are high ITCP connections on a Kafka cluster, it means that there are many connections that have been established but not yet closed. This can cause a number of issues, including decreased performance and potential instability of the cluster. It is important to identify and address these connections in a timely manner to prevent further issues.


This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.


Kafka node underutilization check incident.

This incident type refers to an issue where the network bandwidth on one or more Kafka nodes becomes fully saturated, causing the node(s) to experience performance degradation or even failure. This can result in message delivery delays or loss, and can also impact other nodes that rely on the affected node(s) for message replication. The saturation can be caused by a variety of factors, such as increased message traffic, misconfiguration, or hardware issues.


Network Saturation Check for Kafka Nodes

This incident type refers to a situation where an unauthorized user gains access to a Kafka cluster due to misconfigured Access Control Lists (ACLs) in Kafka. This can result in potential data breaches or other security issues, as the unauthorized user may be able to read, write or modify data within the Kafka cluster. The incident typically requires immediate attention and remediation to prevent further unauthorized access.


Kafka Unauthorized Access

```shell
export KAFKA_PORT="PLACEHOLDER"

export KAFKA_PID="PLACEHOLDER"

export PATH_TO_KAFKA_HOME="PLACEHOLDER"

export CURRENT_NODE_ID="PLACEHOLDER"

export NUMBER_OF_NEW_NODES_TO_ADD="PLACEHOLDER"

export CURRENT_NODE_HOSTNAME="PLACEHOLDER"
```


### Check current number of established ITCP connections

```shell
netstat -anp | grep ${KAFKA_PORT} | grep -c ESTABLISHED
```

### Check the maximum number of allowed ITCP connections

```shell
sysctl net.ipv4.tcp_max_syn_backlog
```

### Check the current number of half-open ITCP connections

```shell
ss -s | grep "TCP:" | awk '{print "Half-open connections: " $4}'
```

### Check the number of sockets in TIME\_WAIT state

```shell
netstat -an | grep ${KAFKA_PORT} | grep TIME_WAIT | wc -l
```

### Check the number of sockets in CLOSE\_WAIT state

```shell
netstat -an | grep ${KAFKA_PORT} | grep CLOSE_WAIT | wc -l
```

### Check the current number of file descriptors used by Kafka processes

```shell
lsof | grep ${KAFKA_PID} | wc -l
```

### Check the current amount of memory used by Kafka processes

```shell
ps -p ${KAFKA_PID} -o rss=
```

### Check the current amount of CPU usage by Kafka processes

```shell
top -p ${KAFKA_PID}
```


### Increase Kafka Cluster capacity: One of the possible reasons for high ITCP connections is that the Kafka cluster is reaching its capacity limit. In this case, we can consider increasing the capacity of the Kafka cluster to handle more connections.

```shell


#!/bin/bash



# Define variables

KAFKA_HOME=${PATH_TO_KAFKA_HOME}

NEW_NODE_COUNT=${NUMBER_OF_NEW_NODES_TO_ADD}



# Stop Kafka services

sudo systemctl stop kafka



# Add new Kafka nodes

for i in $(seq 1 $NEW_NODE_COUNT);

do

    # Create new Kafka node

    sudo mkdir -p $KAFKA_HOME/kafka-node-$i

    sudo chown -R kafka:kafka $KAFKA_HOME/kafka-node-$i



    # Copy Kafka configuration files

    sudo cp -r $KAFKA_HOME/config $KAFKA_HOME/kafka-node-$i/

    sudo cp -r $KAFKA_HOME/logs $KAFKA_HOME/kafka-node-$i/



    # Update Kafka configuration files

    sudo sed -i "s/broker.id=${CURRENT_NODE_ID}/broker.id=$i/" $KAFKA_HOME/kafka-node-$i/config/server.properties

    sudo sed -i "s/#listeners/listeners/" $KAFKA_HOME/kafka-node-$i/config/server.properties

    sudo sed -i "s/#advertised.listeners/advertised.listeners/" $KAFKA_HOME/kafka-node-$i/config/server.properties

    sudo sed -i "s/${CURRENT_NODE_HOSTNAME}/$HOSTNAME/" $KAFKA_HOME/kafka-node-$i/config/server.properties

done



# Start Kafka services

sudo systemctl start kafka


```


High ITCP Connections on Kafka Cluster

Overview

Parameters

Debug

Check current number of established ITCP connections

Check the maximum number of allowed ITCP connections

Check the current number of half-open ITCP connections

Check the number of sockets in TIME_WAIT state

Check the number of sockets in CLOSE_WAIT state

Check the current number of file descriptors used by Kafka processes

Check the current amount of memory used by Kafka processes

Check the current amount of CPU usage by Kafka processes

Repair

Increase Kafka Cluster capacity: One of the possible reasons for high ITCP connections is that the Kafka cluster is reaching its capacity limit. In this case, we can consider increasing the capacity of the Kafka cluster to handle more connections.

Learn more

Related Runbooks

Network Partitioning Between Kafka Brokers.

Parameters

Kafka node underutilization check incident.

Network Saturation Check for Kafka Nodes

Support