Many infrastructure groups deploy Argo for workflow orchestration on Kubernetes. While Argo makes declaratively managing workflows easy, it can leave behind many stale pods after workflow execution.
In Kubernetes, each of these stale pods consumes one IP address, whether it is running or not. Since every Argo pod claims an IP address, you must delete them all eventually. When IPs are exhausted on a node, Kubernetes cannot use any free CPU and memory for scheduling. In an autoscaled environment, this means that Argo IP exhaustion can trigger the provisioning of new capacity prematurely. Most customers overcome this hurdle by either over-provisioning hardware which leads to higher costs or implementing custom logic for cluster auto-scaling and workflow clean-up.