Why Pods Get Stuck in Pending: A Logical Scheduler Failure Catalog

A pod stuck in Pending is one of the most frustrating Kubernetes experiences. The scheduler is running, nodes are healthy, but nothing happens. Understanding why requires understanding how the scheduler thinks — and where configuration logic breaks down.

Overprovisioned

65%+

Workloads using <50% of requested resources

Misconfigurations

40%

Orgs detecting K8s misconfigs (2024)

Need Rightsizing

37%

Orgs with 50%+ workloads to fix

Scale-Up Time

3-4 min

Typical new node provisioning

According to Datadog’s container research, more than 65% of Kubernetes workloads are utilizing less than half of their requested CPU and memory. This overprovisioning creates a paradox: clusters that appear full on paper but are mostly idle in practice — and new pods get stuck Pending because requests, not usage, determine scheduling.

How the Scheduler Decides

The Kubernetes scheduler runs a two-phase algorithm for every unscheduled pod. First, it filters out nodes that cannot possibly run the pod — those lacking sufficient resources, having incompatible taints, or failing to match node selectors. Then it scores the remaining nodes to find the best fit, considering factors like resource balance, affinity preferences, and image locality.

If every node gets filtered out, the pod stays Pending. The scheduler checks for unschedulable pods every 10 seconds, but if there’s no node that passes all predicates, no amount of waiting helps.

Requests vs. Usage

Kubernetes schedules based on resource requests, not actual usage. A node with 4 CPU that has 3.8 CPU requested cannot accept a pod requesting 500m — even if actual usage is only 1 CPU. This is intentional: requests are guarantees, and the scheduler must honor them.

The Seven Reasons Pods Stay Pending

1. Insufficient CPU or Memory

The most common cause. Every node has some CPU or memory already reserved by existing pod requests, and the new pod’s requests push the total over allocatable capacity.

The scheduler reports something like: 0/5 nodes are available: 5 Insufficient cpu.

What’s actually happening: the new pod requests 2 CPU, but every node in the cluster has less than 2 CPU remaining after subtracting existing requests from allocatable capacity. The math that matters is Available = Allocatable - Sum(Pod Requests), not Available = Allocatable - Actual Usage.

To diagnose, run kubectl describe nodes and check the “Allocated resources” section. Compare requested resources against allocatable capacity. Then identify which pods are requesting the most with kubectl get pods -A and examine their resource specifications.

Symptom	Root Cause	Fix
Insufficient cpu	Pod requests exceed remaining capacity	Reduce requests or add nodes
Insufficient memory	Memory requests exceed remaining capacity	Reduce requests or add nodes
Immediate failure on scale	Cluster at request capacity despite low usage	Right-size existing workloads

According to Fairwinds’ 2024 Kubernetes Benchmark Report, 37% of organizations have 50% or more workloads in need of container rightsizing. This overprovisioning directly contributes to scheduling failures — capacity exists, but it’s locked up in inflated requests.

2. Unsatisfiable Node Selectors

The pod specifies a nodeSelector that doesn’t match any node’s labels. Common mistakes include typos (disktype vs disk-type), case sensitivity issues (Environment: Production vs environment: production), or requesting labels that simply don’t exist (like gpu: "true" when no GPU nodes are present).

The scheduler reports: 0/5 nodes are available: 5 node(s) didn't match Pod's node affinity/selector.

To diagnose, run kubectl get nodes --show-labels to see what labels nodes actually have, then compare against the pod’s nodeSelector with kubectl get pod <name> -o yaml.

3. Taint/Toleration Mismatch

Nodes can be tainted to repel pods that don’t explicitly tolerate the taint. This is commonly used for master nodes (node-role.kubernetes.io/control-plane:NoSchedule), GPU nodes reserved for specific workloads, or nodes under maintenance.

The scheduler reports: 0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.

A pod must include a matching toleration for every taint on a node. If any taint isn’t tolerated, the pod won’t schedule there. Run kubectl describe nodes | grep Taints to see what taints exist, then verify the pod’s tolerations match.

Hidden Taints

Control plane nodes are tainted by default. If you’re running a small cluster and accidentally exclude workers, pods may have nowhere to go. Also watch for cloud providers adding taints during node maintenance or spot instance interruption warnings.

4. Affinity Deadlocks

Pod affinity rules can create circular dependencies. Imagine Pod A requires being co-located with Pod B, and Pod B requires being co-located with Pod A. Neither can schedule first, so both stay Pending forever.

Anti-affinity can also overreach. A common pattern is requiring that no two replicas of a service run on the same node. This works fine with 5 replicas on 5 nodes. Try to scale to 6 replicas on 5 nodes, and the sixth pod has nowhere to go.

The scheduler reports something like: 0/5 nodes are available: 2 node(s) didn't match pod affinity rules, 3 node(s) didn't match pod anti-affinity rules.

The fix is often switching from requiredDuringSchedulingIgnoredDuringExecution (hard requirement) to preferredDuringSchedulingIgnoredDuringExecution (soft preference). Soft affinity lets the scheduler make best-effort placement rather than failing entirely.

5. PersistentVolumeClaim Not Bound

If a pod references a PVC that isn’t bound to a PersistentVolume, the pod can’t start. Common causes include requesting a non-existent storage class, requesting more storage than any PV offers, or zone mismatches where the volume exists in one availability zone but the pod can only schedule in another.

The scheduler reports: 0/5 nodes are available: 5 node(s) didn't find available persistent volumes to bind.

Run kubectl get pvc to check binding status and kubectl describe pvc <name> for detailed events. In cloud environments, pay special attention to zone topology — a volume in us-east-1a can’t be mounted by a pod on a node in us-east-1b.

6. Resource Quota Exceeded

Namespace-level ResourceQuotas limit total resources that can be requested. Even if nodes have capacity, a pod won’t be created if it would push the namespace over quota.

The error appears at pod creation: Error creating: pods "my-pod" is forbidden: exceeded quota: compute-resources, requested: cpu=500m, used: 1900m, limited: 2

Run kubectl describe quota -n <namespace> to see current usage versus limits. Either reduce requests in existing pods, delete unused pods, or request a quota increase.

7. Cluster Autoscaler Lag

When all nodes are at capacity and cluster autoscaling is enabled, the autoscaler must provision new nodes. According to AWS documentation, typical node provisioning takes 3-4 minutes. During this window, pods remain Pending.

The Cluster Autoscaler FAQ notes that the autoscaler checks for unschedulable pods every 10 seconds and issues scale-up requests within 30-60 seconds. But the cloud provider then needs time to create and register the node — sometimes up to 6 minutes in complex environments.

If pods are Pending longer than expected, check autoscaler logs. The autoscaler might be at its node limit, lack permissions to provision nodes, or be waiting for node groups that are already at maximum capacity.

Logical vs. Physical Failures

Notice that none of these are physical failures. The nodes are healthy. The scheduler is running. The network is fine. The problem is purely logical — a mismatch between what the pod requires and what the cluster can provide under its current configuration.

Failure Type	Infrastructure Status	Fix Category
Insufficient CPU/Memory	Healthy	Configuration or capacity
Node selector mismatch	Healthy	Configuration
Taint not tolerated	Healthy	Configuration
Affinity deadlock	Healthy	Configuration
PVC unbound	Healthy	Storage configuration
Quota exceeded	Healthy	Policy/configuration

This distinction matters for incident response. Pending pods don’t need node repairs or network debugging — they need configuration review and capacity planning.

Preventing Scheduling Failures

Use LimitRanges for defaults. Without explicit requests, pods can be scheduled without reserving resources, leading to overcommitment and later eviction. A LimitRange ensures every container has at least minimum requests set, making scheduling behavior predictable.

Prefer soft affinity over hard requirements. Use preferredDuringSchedulingIgnoredDuringExecution instead of required variants. Soft affinity lets the scheduler make tradeoffs when the ideal placement isn’t available, rather than leaving pods Pending indefinitely.

Right-size your requests. The Fairwinds benchmark shows that 57% of organizations have gotten 90% or more of their workloads properly sized — but the remaining organizations are leaving significant capacity locked in overprovisioned requests. Use tools like Vertical Pod Autoscaler in recommendation mode, or observability platforms that track actual usage versus requests.

Alert on extended Pending states. A pod Pending for 30 seconds during a scale-up is normal. A pod Pending for 10 minutes indicates a problem. Set up monitoring with a query like kube_pod_status_phase{phase="Pending"} joined with pod creation time to catch pods that aren’t progressing.

Quick Diagnostic Sequence

When a pod is stuck Pending: first, run kubectl describe pod <name> and read the Events section — it usually tells you exactly why. Second, check node capacity with kubectl describe nodes. Third, verify any referenced PVCs are bound. This three-step sequence resolves most cases in under a minute.

The Bottom Line

Pending pods are configuration failures, not infrastructure failures. The scheduler is doing exactly what it’s supposed to do — refuse to place pods on nodes that can’t satisfy their requirements. The fix is almost always in the pod spec, the node labels, the resource requests, or the cluster capacity.

Understanding this helps prioritize the response. You don’t need to page someone at 2 AM for a node selector typo. You do need capacity planning reviews if pods are frequently Pending due to resource exhaustion. And you need to audit affinity rules if complex scheduling requirements are creating deadlocks.

References

This is part 3 of our “Kubernetes Failure Catalog” series. Next: CrashLoopBackOff Is a Symptom.