> For the complete documentation index, see [llms.txt](https://docs.acecloud.ai/knowledge-base/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.acecloud.ai/knowledge-base/kubernetes-hosted/troubleshooting-kubernetes-cluster.md).

# Troubleshooting Kubernetes Cluster

## Overview

Many apparent platform issues are caused by a specific configuration detail. This guide is organised by symptom. Please find the section that matches what you are seeing and review the likely causes before contacting support.

### My PersistentVolumeClaim is stuck in "Pending"

| **Possible cause**                                  | **Details**                                                                                                                                    |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Incorrect or misspelled StorageClass name           | Please run kubectl get storageclass and confirm that the name matches your PVC.                                                                |
| ReadWriteMany requested on the default StorageClass | acecloud-block-sc supports ReadWriteOnce only. Please see our guide on provisioning cloud resources for shared-access alternatives.            |
| No pod is using the volume yet                      | With WaitForFirstConsumer binding mode, the volume is not provisioned until a pod that references it is scheduled. This is expected behaviour. |
| Storage quota reached                               | Please check usage on the AceCloud dashboard. A full quota can produce the same symptom as a misconfigured provisioner.                        |

### I deleted a PersistentVolumeClaim and the data was lost

This is expected behaviour when the default reclaim policy is Delete. Deleting a PVC deletes both the PV object and the underlying volume in your project. There is nothing further to clean up, but the data is not recoverable. If volumes need to survive their claim, please use a StorageClass with reclaimPolicy: Retain. Always confirm the reclaim policy before deleting anything you need to keep.

### I deleted a PersistentVolumeClaim with a Retain policy, but the volume still shows up in my project

This is expected and is not a leftover error. Retain only affects Kubernetes. Deleting the PVC and the resulting PV object removes them from the cluster, but it does not delete the underlying volume from your AceCloud project. Please delete the volume separately from the portal:

| **Step** | **Action**                                                                                  |
| -------- | ------------------------------------------------------------------------------------------- |
| 1        | Run kubectl get pv and note the volume name for the relevant PV.                            |
| 2        | In the AceCloud portal, go to **Storage**, then **Volumes**, and search for that same name. |
| 3        | Delete the volume from the portal.                                                          |

Until you do this, the volume continues to exist in your project and continues to count towards your storage quota and cost, even though it no longer appears in kubectl.

### I deleted a StatefulSet, but its storage still exists and continues to incur costs

This is expected: StatefulSet PVCs are not deleted automatically with the StatefulSet. Once the PVCs are no longer needed, please delete them directly. If they use a Retain StorageClass, please also delete the corresponding volumes from **Storage**, then **Volumes** in the portal, as described above. Otherwise, they will continue to incur costs even after the PVC is gone.

### Resizing my PersistentVolumeClaim had no effect

Please confirm that the StorageClass has allowVolumeExpansion: true. Without this setting, a size change may be accepted by the API but not applied.

### Large file or database dump transfers via kubectl cp or kubectl exec fail partway through

These commands route data through the Kubernetes API server rather than transferring it directly, and they are not designed for large or long-running transfers. A mid-transfer timeout can leave an incomplete file without a clear error, which is often reported as a platform issue even though the transfer method is the cause.

#### Recommended alternatives

| **Option**                             | **Details**                                                                                                                                 |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Transfer within the cluster network    | Run a temporary pod with the target volume mounted and use rsync or scp directly to it.                                                     |
| Stage the data in object storage first | Then pull it into the volume with curl, wget, or your provider's CLI.                                                                       |
| Use a Kubernetes Job                   | This pulls the source data directly, so the transfer does not depend on a persistent connection from your workstation.                      |
| Use Velero                             | For backup, restore, or cluster-to-cluster migration of volume data, Velero is usually more reliable than manual copying for this use case. |

### A node is not scaling down

In many cases, the node simply needs more time. A node must remain underused for approximately 10 minutes before it is considered for scale-down, and scale-down pauses for approximately another 10 minutes after any recent scale-up.

If it has been longer than that, please check whether resource requests are set well above actual usage. Overestimated requests can prevent a node from appearing underused.

#### A node is not scaling down, and I have a PodDisruptionBudget configured

Node scale-down evicts pods, and eviction respects PodDisruptionBudgets. A PDB set to maxUnavailable: 0, or minAvailable equal to the total replica count, blocks any eviction and therefore blocks scale-down indefinitely. This is intentional behaviour rather than a malfunction.

To resolve this, please set the PDB to allow at least one pod of disruption relative to your replica count, for example maxUnavailable: 1, or minAvailable strictly below the replica count.

### The cluster is not scaling up despite apparent high load

Please check that pods have resource requests set. Autoscaling responds to pods that cannot be scheduled because of requested resources. It does not respond to real-time usage alone, so missing requests usually mean there is no scale-up trigger.

Please also check your node pool maximum and your project's quota. If either limit has been reached, the behaviour may look like autoscaling is stuck, but the cluster is operating at its configured ceiling.

Autoscaling is reactive, not predictive. It will not scale ahead of expected demand, so please scale manually for planned spikes.

### A node became unresponsive or went down under load

This is usually caused by a combination of factors: HPA increases the replica count, cluster autoscaling is disabled or already at its maximum, and the extra replicas are placed on existing nodes. If actual usage exceeds requested resources, or memory limits are set far above requests and allow overcommit, total usage can exceed the node's real capacity. This can cause memory pressure and, in severe cases, a NotReady node. Because HPA visibly worked and the failure appears afterwards, this is often reported as a platform issue rather than a sizing problem.

#### To avoid this

| **Action**                                           | **Reason**                                                         |
| ---------------------------------------------------- | ------------------------------------------------------------------ |
| Enable cluster autoscaling                           | Set a maximum node count that can absorb the full HPA scale range. |
| Set resource requests to match actual observed usage | A conservative underestimate can lead to overcommit.               |
| Keep memory limits close to requests                 | A large gap can allow overcommit.                                  |
| Use HPA together with cluster autoscaling            | HPA alone may not provide enough node capacity.                    |

### My Service has no external IP address

Each LoadBalancer Service uses part of your project's load balancer quota. Please check usage on the AceCloud dashboard if you have created several LoadBalancer Services. For many services, an Ingress controller routing through a single load balancer is usually the better fit.

### My application experienced downtime during a routine cluster operation (node drain, upgrade, etc.)

This can be expected if the application is running a single replica with no PodDisruptionBudget. Node drains are a normal cluster operation, and a single unprotected replica has nothing to prevent disruption. To avoid this, please run multiple replicas with an appropriate PDB.

### I have access through the AceCloud portal, but kubectl returns "Forbidden"

Portal access, including downloading a KubeConfig, does not grant cluster permissions on its own:

| **Fact**                                                                          | **Detail**                                                                                        |
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| AceCloud automatically grants access to the Admins group and the cluster creator. | Anyone else needs an explicit Access Rule for that specific cluster, regardless of portal access. |

If you are receiving a Forbidden response despite successful sign-in, please confirm with your administrator whether you have an Access Rule for that cluster. See our RBAC configuration guide for details.

### An assigned access policy does not grant the permissions I expected

Policy names do not always match their actual scope. Please open the policy detail view in the portal to check its actual permissions before assigning it.

### I revoked a user's access, but they can still run commands

This can take up to 15 minutes, as the change applies on the next token refresh. For immediate effect, please revoke the user's KubeConfig from IAM directly rather than only removing the Access Rule.

**Still need assistance?**

If the relevant section above does not resolve the issue, please contact AceCloud Support with the specific resource involved, such as the PVC, node pool, Service, or Access Rule, and its current status from kubectl describe or the portal. This helps us quickly distinguish a configuration issue from a platform-side issue.

## **Related Articles**

[Using Cloud Resources](https://docs.acecloud.ai/knowledge-base/kubernetes-hosted/using-cloud-resources-in-acecloud-hosted-kubernetes#cluster-autoscaling)