Problem For several Kubernetes pod-level faults (for example pod memory hog, CPU hog, network, DNS, disk fill, container kill), Harness Chaos creates a short-lived helper pod that must run on the same node as the target application pod so it can use that node’s container runtime socket.
If the target node has already hit its max pods per node limit (example: used 80 / capacity 80), the helper is rejected with OutOfpods. GKE Cluster Autoscaler does not reliably fix this, because:
This is a pod count limit, not a CPU/memory shortage.
The helper is pinned to the target node, so a newly scaled node elsewhere cannot host it.
Customer impact A large GKE customer plans to run limited Chaos experiments as part of pre-prod deployment pipelines. Application pods can land on any node, including nearly full ones. They cannot guarantee spare pod slots for helper pods at experiment time, so pipelines can fail due to node packing rather than application behaviour.
Requested capability Add an optional persistent per-node Chaos component (DaemonSet-style proxy/agent on every node) that already has access to the local container runtime.
At experiment time, Chaos would instruct the proxy already running on the target node to inject the fault, instead of scheduling a new helper pod onto that same node.
Benefits:
Avoids last-minute helper scheduling failures on dense nodes
Lets orchestration/control pods schedule where capacity exists (including after node pool autoscaling)
Makes Chaos safer to run in CI/CD / pre-prod pipelines on densely packed clusters
Trade-off the customer accepts: one always-on pod per node, plus the cost/ops of managing that DaemonSet.
Ask Please evaluate this as a product enhancement for Kubernetes Chaos (Harness Infrastructure / DDCR). Customer is happy to provide more environment detail and validate designs.
Created by Aadesh Bhardwaj
·