---
id: d2326123-2c5e-4ccc-b2a7-353181d9e66d
---
# Troubleshooting connectivity issues between tasks in an Amazon ECS cluster using service discovery
---

This incident type involves connectivity issues between tasks in an Amazon ECS (Elastic Container Service) cluster that uses service discovery. Service discovery is a mechanism that allows services to be discovered and accessed by other services without needing to know their IP addresses. There are several potential areas to investigate when facing connectivity issues, including service discovery configuration, DNS resolution, task definition and network mode, security groups, task IAM role, VPC configuration, ECS agent, ECS service event messages, logs, application-level configuration, and health checks. Troubleshooting steps need to be taken to resolve these issues.

### Parameters
```shell
export SERVICE_DISCOVERY_SERVICE_ID="PLACEHOLDER"

export TASK_DEFINITION_ARN="PLACEHOLDER"

export TASK_ARN="PLACEHOLDER"

export CLUSTER_NAME="PLACEHOLDER"

export VPC_ID="PLACEHOLDER"
```

## Debug

### Confirm that the ECS service is associated with a Service Discovery namespace
```shell
aws servicediscovery get-service --id ${SERVICE_DISCOVERY_SERVICE_ID}
```

### Check if the DNS records of the tasks are correctly registered in the AWS Cloud Map service
```shell
aws servicediscovery list-instances --service-id ${SERVICE_DISCOVERY_SERVICE_ID}
```

### Review the task definition and check the network mode
```shell
aws ecs describe-task-definition --task-definition ${TASK_DEFINITION_ARN} 
```

### Confirm that the tasks are launched in the expected subnets
```shell
aws ecs describe-tasks --tasks ${TASK_ARN} --cluster ${CLUSTER_NAME}
```

### Ensure health checks are correctly set up
```shell
#!/bin/bash



for service_arn in $(aws ecs list-services --cluster ${CLUSTER_NAME} --query 'serviceArns[]' --output text);

do

    service_task_def=$(aws ecs describe-services --cluster ${CLUSTER_NAME} --services $service_arn --query 'services[0].taskDefinition' --output text)

    service_tasks=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --service-name $service_arn --query 'taskArns[]' --output text)

    if [[ $service_task_def == ${TASK_DEFINITION_ARN} ]] && [[ $service_tasks == ${TASK_ARN} ]]; then

        service_arn=$service_arn

    fi

done



aws ecs describe-services --services $service_arn --cluster ${CLUSTER_NAME}
```

### Review security groups associated with the task or service to make sure inbound and outbound traffic is allowed between tasks
```shell
#!/bin/bash



for service_arn in $(aws ecs list-services --cluster ${CLUSTER_NAME} --query 'serviceArns[]' --output text);

do

    service_task_def=$(aws ecs describe-services --cluster ${CLUSTER_NAME} --services $service_arn --query 'services[0].taskDefinition' --output text)

    service_tasks=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --service-name $service_arn --query 'taskArns[]' --output text)

    if [[ $service_task_def == ${TASK_DEFINITION_ARN} ]] && [[ $service_tasks == ${TASK_ARN} ]]; then

        service_arn=$service_arn

    fi

done



security_group_id=$(aws ecs describe-services --services $service_arn --cluster ${CLUSTER_NAME} --query 'services[0].networkConfiguration.awsvpcConfiguration.securityGroups[]' --output text)



aws ec2 describe-security-groups --group-ids $security_group_id
```

## Repair

### Associate the service with the service registries
```shell


#!/bin/bash



for service_arn in $(aws ecs list-services --cluster ${CLUSTER_NAME} --query 'serviceArns[]' --output text);

do

    service_task_def=$(aws ecs describe-services --cluster ${CLUSTER_NAME} --services $service_arn --query 'services[0].taskDefinition' --output text)

    service_tasks=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --service-name $service_arn --query 'taskArns[]' --output text)

    if [[ $service_task_def == ${TASK_DEFINITION_ARN} ]] && [[ $service_tasks == ${TASK_ARN} ]]; then

        ECS_SERVICE=$service_arn

    fi

done





service_discovery_service_arn=$(aws servicediscovery get-service --id ${SERVICE_DISCOVERY_SERVICE_ID} --query "Service.Arn" --output text)



# Check the ECS service's configuration

aws ecs describe-services --services $ECS_SERVICE







# If the service is not associated with a Service Discovery namespace, associate it with one

if [[ ! $(aws ecs describe-services --services $ECS_SERVICE --query 'services[0].serviceRegistries[0].registryArn' --output text) ]]; then

    aws ecs update-service --service $ECS_SERVICE --service-registries registryArn=$service_discovery_service_arn

fi


```

### Enable DNS resolution for the VPC
```shell


#!/bin/bash



# Set variables

VPC_ID=${VPC_ID}



# Enable DNS resolution for the VPC

aws ec2 modify-vpc-attribute --vpc-id $VPC_ID --enable-dns-support "{\"Value\":true}"
```


This incident type involves connectivity issues between tasks in an Amazon ECS (Elastic Container Service) cluster that uses service discovery. Service discovery is a mechanism that allows services to be discovered and accessed by other services without needing to know their IP addresses. There are several potential areas to investigate when facing connectivity issues, including service discovery configuration, DNS resolution, task definition and network mode, security groups, task IAM role, VPC configuration, ECS agent, ECS service event messages, logs, application-level configuration, and health checks. Troubleshooting steps need to be taken to resolve these issues.


```shell
export SERVICE_DISCOVERY_SERVICE_ID="PLACEHOLDER"

export TASK_DEFINITION_ARN="PLACEHOLDER"

export TASK_ARN="PLACEHOLDER"

export CLUSTER_NAME="PLACEHOLDER"

export VPC_ID="PLACEHOLDER"
```


### Confirm that the ECS service is associated with a Service Discovery namespace

```shell
aws servicediscovery get-service --id ${SERVICE_DISCOVERY_SERVICE_ID}
```

### Check if the DNS records of the tasks are correctly registered in the AWS Cloud Map service

```shell
aws servicediscovery list-instances --service-id ${SERVICE_DISCOVERY_SERVICE_ID}
```

### Review the task definition and check the network mode

```shell
aws ecs describe-task-definition --task-definition ${TASK_DEFINITION_ARN} 
```

### Confirm that the tasks are launched in the expected subnets

```shell
aws ecs describe-tasks --tasks ${TASK_ARN} --cluster ${CLUSTER_NAME}
```

### Ensure health checks are correctly set up

```shell
#!/bin/bash



for service_arn in $(aws ecs list-services --cluster ${CLUSTER_NAME} --query 'serviceArns[]' --output text);

do

    service_task_def=$(aws ecs describe-services --cluster ${CLUSTER_NAME} --services $service_arn --query 'services[0].taskDefinition' --output text)

    service_tasks=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --service-name $service_arn --query 'taskArns[]' --output text)

    if [[ $service_task_def == ${TASK_DEFINITION_ARN} ]] && [[ $service_tasks == ${TASK_ARN} ]]; then

        service_arn=$service_arn

    fi

done



aws ecs describe-services --services $service_arn --cluster ${CLUSTER_NAME}
```

### Review security groups associated with the task or service to make sure inbound and outbound traffic is allowed between tasks

```shell
#!/bin/bash



for service_arn in $(aws ecs list-services --cluster ${CLUSTER_NAME} --query 'serviceArns[]' --output text);

do

    service_task_def=$(aws ecs describe-services --cluster ${CLUSTER_NAME} --services $service_arn --query 'services[0].taskDefinition' --output text)

    service_tasks=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --service-name $service_arn --query 'taskArns[]' --output text)

    if [[ $service_task_def == ${TASK_DEFINITION_ARN} ]] && [[ $service_tasks == ${TASK_ARN} ]]; then

        service_arn=$service_arn

    fi

done



security_group_id=$(aws ecs describe-services --services $service_arn --cluster ${CLUSTER_NAME} --query 'services[0].networkConfiguration.awsvpcConfiguration.securityGroups[]' --output text)



aws ec2 describe-security-groups --group-ids $security_group_id
```


### Associate the service with the service registries

```shell


#!/bin/bash



for service_arn in $(aws ecs list-services --cluster ${CLUSTER_NAME} --query 'serviceArns[]' --output text);

do

    service_task_def=$(aws ecs describe-services --cluster ${CLUSTER_NAME} --services $service_arn --query 'services[0].taskDefinition' --output text)

    service_tasks=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --service-name $service_arn --query 'taskArns[]' --output text)

    if [[ $service_task_def == ${TASK_DEFINITION_ARN} ]] && [[ $service_tasks == ${TASK_ARN} ]]; then

        ECS_SERVICE=$service_arn

    fi

done





service_discovery_service_arn=$(aws servicediscovery get-service --id ${SERVICE_DISCOVERY_SERVICE_ID} --query "Service.Arn" --output text)



# Check the ECS service's configuration

aws ecs describe-services --services $ECS_SERVICE







# If the service is not associated with a Service Discovery namespace, associate it with one

if [[ ! $(aws ecs describe-services --services $ECS_SERVICE --query 'services[0].serviceRegistries[0].registryArn' --output text) ]]; then

    aws ecs update-service --service $ECS_SERVICE --service-registries registryArn=$service_discovery_service_arn

fi


```

### Enable DNS resolution for the VPC

```shell


#!/bin/bash



# Set variables

VPC_ID=${VPC_ID}



# Enable DNS resolution for the VPC

aws ec2 modify-vpc-attribute --vpc-id $VPC_ID --enable-dns-support "{\"Value\":true}"
```


Troubleshooting connectivity issues between tasks in an Amazon ECS cluster using service discovery

Overview

Parameters

Debug

Confirm that the ECS service is associated with a Service Discovery namespace

Check if the DNS records of the tasks are correctly registered in the AWS Cloud Map service

Review the task definition and check the network mode

Confirm that the tasks are launched in the expected subnets

Ensure health checks are correctly set up

Review security groups associated with the task or service to make sure inbound and outbound traffic is allowed between tasks

Repair

Associate the service with the service registries

Enable DNS resolution for the VPC

Learn more

Related Runbooks

Support