---
id: 349f05e3-50fd-4560-84b3-9cb473ad851b
---

# Application Load Balancer Health Check Failure During Green Deployment for AWS ECS Fargate
---

This incident type involves ECS tasks failing Application Load Balancer health checks during a new green deployment in an AWS ECS Fargate environment. In such a scenario, the issue may be caused by multiple ECS services trying to register their tasks to the same green target group, leading to a discrepancy. The recommended troubleshooting step is to update the load balancer configuration to ensure that only one ECS service or port is registered to one target group.

### Parameters
```shell
export CLUSTER_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export TASK_DEFINITION_ARN="PLACEHOLDER"

export LOAD_BALANCER_ARN="PLACEHOLDER"

export TARGET_GROUP_ARN="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export CONTAINER_PORT="PLACEHOLDER"
```

## Debug

### List all the available services in the cluster
```shell
aws ecs list-services --cluster ${CLUSTER_NAME}
```

### Describe the specified ECS service to check its details
```shell
aws ecs describe-services --cluster ${CLUSTER_NAME} --services ${SERVICE_NAME}
```

### Describe the specified task definition
```shell
aws ecs describe-task-definition --task-definition ${TASK_DEFINITION_ARN}
```

### List all the registered target groups of the specified load balancer
```shell
aws elbv2 describe-target-groups --load-balancer-arn ${LOAD_BALANCER_ARN}
```

### List all the tasks registered to the specified target group
```shell
#!/bin/bash



IPs=$(aws elbv2 describe-target-health --target-group-arn ${TARGET_GROUP_ARN} --query 'TargetHealthDescriptions[].Target.Id' --output text)



# For each target IP

for IP in $IPs; do

    # List all task ARNs in the cluster

    TASK_ARNS=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --query "taskArns[]" --output text)



    # For each task ARN, describe the task to get its network configuration

    for TASK_ARN in $TASK_ARNS; do

        TASK_IP=$(aws ecs describe-tasks --cluster ${CLUSTER_NAME} --tasks $TASK_ARN --query "tasks[0].attachments[0].details[?name=='privateIPv4Address'].value" --output text)

        

        # If the task IP matches the target IP, print it out

        if [[ "$TASK_IP" == "$IP" ]]; then

            TASK_ARN=$TASK_ARN

        fi

    done

done





aws ecs describe-tasks --cluster ${CLUSTER_NAME} --tasks $TASK_ARN
```

## Repair

### Update the load balancer configuration to ensure that only one ECS service or port is registered to one target group.
```shell


#!/bin/bash



# Set variables

CLUSTER=${CLUSTER_NAME}

TARGET_GROUP_ARN=${TARGET_GROUP_ARN}

CONTAINER_NAME=${CONTAINER_NAME}

CONTAINER_PORT=${CONTAINER_PORT}



# Deregister all targets from the target group

aws elbv2 deregister-targets --target-group-arn $TARGET_GROUP_ARN --targets Id=$(aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN | jq -r '.TargetHealthDescriptions[].Target.Id')



# Update the ECS service 

aws ecs update-service --cluster $CLUSTER --service $SERVICE --load-balancer "[{\"containerName\": \"$CONTAINER_NAME\", \"conatinerPort\": $CONTAINER_PORT, \"targetGroupArn\": \"$TARGET_GROUP_ARN\"}]"



echo "Target group configuration updated successfully."


```

### Check the current status of the ECS Service and tasks to determine whether they are running correctly.
```shell


#!/bin/bash



# Assign Variables

CLUSTER_NAME=${CLUSTER_NAME}

SERVICE_NAME=${SERVICE_NAME}



# Check Status of ECS Cluster

echo "Checking status of ECS Cluster: $CLUSTER_NAME"

cluster_status=$(aws ecs describe-clusters --cluster $CLUSTER_NAME --query "clusters[0].status" --output text)

echo "Cluster Status: $cluster_status"



# Check Status of ECS Service

echo "Checking status of ECS Service: $SERVICE_NAME"

service_status=$(aws ecs describe-services --cluster $CLUSTER_NAME --services $SERVICE_NAME --query "services[0].status" --output text)

echo "Service Status: $service_status"


```

This incident type involves ECS tasks failing Application Load Balancer health checks during a new green deployment in an AWS ECS Fargate environment. In such a scenario, the issue may be caused by multiple ECS services trying to register their tasks to the same green target group, leading to a discrepancy. The recommended troubleshooting step is to update the load balancer configuration to ensure that only one ECS service or port is registered to one target group.


```shell
export CLUSTER_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export TASK_DEFINITION_ARN="PLACEHOLDER"

export LOAD_BALANCER_ARN="PLACEHOLDER"

export TARGET_GROUP_ARN="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export CONTAINER_PORT="PLACEHOLDER"
```


### List all the available services in the cluster

```shell
aws ecs list-services --cluster ${CLUSTER_NAME}
```

### Describe the specified ECS service to check its details

```shell
aws ecs describe-services --cluster ${CLUSTER_NAME} --services ${SERVICE_NAME}
```

### Describe the specified task definition

```shell
aws ecs describe-task-definition --task-definition ${TASK_DEFINITION_ARN}
```

### List all the registered target groups of the specified load balancer

```shell
aws elbv2 describe-target-groups --load-balancer-arn ${LOAD_BALANCER_ARN}
```

### List all the tasks registered to the specified target group

```shell
#!/bin/bash



IPs=$(aws elbv2 describe-target-health --target-group-arn ${TARGET_GROUP_ARN} --query 'TargetHealthDescriptions[].Target.Id' --output text)



# For each target IP

for IP in $IPs; do

    # List all task ARNs in the cluster

    TASK_ARNS=$(aws ecs list-tasks --cluster ${CLUSTER_NAME} --query "taskArns[]" --output text)



    # For each task ARN, describe the task to get its network configuration

    for TASK_ARN in $TASK_ARNS; do

        TASK_IP=$(aws ecs describe-tasks --cluster ${CLUSTER_NAME} --tasks $TASK_ARN --query "tasks[0].attachments[0].details[?name=='privateIPv4Address'].value" --output text)

        

        # If the task IP matches the target IP, print it out

        if [[ "$TASK_IP" == "$IP" ]]; then

            TASK_ARN=$TASK_ARN

        fi

    done

done





aws ecs describe-tasks --cluster ${CLUSTER_NAME} --tasks $TASK_ARN
```


### Update the load balancer configuration to ensure that only one ECS service or port is registered to one target group.

```shell


#!/bin/bash



# Set variables

CLUSTER=${CLUSTER_NAME}

TARGET_GROUP_ARN=${TARGET_GROUP_ARN}

CONTAINER_NAME=${CONTAINER_NAME}

CONTAINER_PORT=${CONTAINER_PORT}



# Deregister all targets from the target group

aws elbv2 deregister-targets --target-group-arn $TARGET_GROUP_ARN --targets Id=$(aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN | jq -r '.TargetHealthDescriptions[].Target.Id')



# Update the ECS service 

aws ecs update-service --cluster $CLUSTER --service $SERVICE --load-balancer "[{\"containerName\": \"$CONTAINER_NAME\", \"conatinerPort\": $CONTAINER_PORT, \"targetGroupArn\": \"$TARGET_GROUP_ARN\"}]"



echo "Target group configuration updated successfully."


```

### Check the current status of the ECS Service and tasks to determine whether they are running correctly.

```shell


#!/bin/bash



# Assign Variables

CLUSTER_NAME=${CLUSTER_NAME}

SERVICE_NAME=${SERVICE_NAME}



# Check Status of ECS Cluster

echo "Checking status of ECS Cluster: $CLUSTER_NAME"

cluster_status=$(aws ecs describe-clusters --cluster $CLUSTER_NAME --query "clusters[0].status" --output text)

echo "Cluster Status: $cluster_status"



# Check Status of ECS Service

echo "Checking status of ECS Service: $SERVICE_NAME"

service_status=$(aws ecs describe-services --cluster $CLUSTER_NAME --services $SERVICE_NAME --query "services[0].status" --output text)

echo "Service Status: $service_status"


```


Application Load Balancer Health Check Failure During Green Deployment for AWS ECS Fargate

Overview

Parameters

Debug

List all the available services in the cluster

Describe the specified ECS service to check its details

Describe the specified task definition

List all the registered target groups of the specified load balancer

List all the tasks registered to the specified target group

Repair

Update the load balancer configuration to ensure that only one ECS service or port is registered to one target group.

Check the current status of the ECS Service and tasks to determine whether they are running correctly.

Learn more

Related Runbooks

Support