Helm

Helm: UPGRADE FAILED and releases stuck pending

A Helm upgrade failed and now the release will not budge. How to recover a pending-upgrade release, why CRDs break upgrades, and what --atomic prevents.

medium fix7 min read

helm. The error
Error: UPGRADE FAILED: another operation (install/upgrade/rollback)
is in progress

Error: UPGRADE FAILED: cannot patch "payments-api" with kind Deployment:
Deployment.apps "payments-api" is invalid: spec.selector: Invalid value:
field is immutable

Do this first3 steps

Run these in order. Each one tells you what its output means before you change anything.

  1. 1

    See what state the release is stuck in

    helm list -a -n payments

    pending-upgrade or pending-install means a previous run was interrupted and left the lock behind. failed means the upgrade ran and the resources never became ready.

  2. 2

    Get back to something that works

    helm history payments-api -n payments

    Find the most recent revision with status deployed and roll back to it with helm rollback payments-api <revision> -n payments. This is almost always the right first move.

  3. 3

    If it is stuck pending and no Helm process is running

    kubectl get secret -n payments -l owner=helm,name=payments-api

    Delete the newest sh.helm.release.v1.* secret, which is the interrupted revision. Only do this once you are certain no helm process is still in flight. Otherwise you are racing it.

All 8 sections

A failed Helm upgrade leaves the release in a state that blocks every subsequent operation. The message you get depends on how it failed, and each one has a different recovery.

Find out what state the release is actually in

helm list -a -n payments

The -a matters, without it, releases in a failed or pending state are hidden, which is exactly when you need to see them.

NAME           NAMESPACE   REVISION   STATUS            CHART
payments-api   payments    7          pending-upgrade   payments-1.5.0
helm history payments-api -n payments
REVISION  STATUS            CHART            DESCRIPTION
5         superseded        payments-1.3.0   Upgrade complete
6         deployed          payments-1.4.0   Upgrade complete
7         pending-upgrade   payments-1.5.0   Preparing upgrade

Revision 6 is the last good one. Revision 7 never completed.

StatusMeaning
deployedHealthy
failedThe upgrade ran and failed (retryable)
pending-upgradeInterrupted mid-operation (blocks everything)
pending-installFirst install interrupted
pending-rollbackA rollback was interrupted

"another operation is in progress"

The release is stuck in a pending-* state, almost always because the process was killed: a cancelled CI job, a closed laptop, a timeout.

First check nothing is genuinely running, then roll forward to the last good revision:

helm rollback payments-api 6 -n payments

That is the clean fix and usually works.

If rollback itself refuses with the same error, Helm's release record needs clearing. Helm 3 stores each revision as a Secret in the release's namespace:

kubectl get secret -n payments -l owner=helm,name=payments-api
NAME                                 TYPE                 AGE
sh.helm.release.v1.payments-api.v6   helm.sh/release.v1   4d
sh.helm.release.v1.payments-api.v7   helm.sh/release.v1   20m

Deleting the pending revision's Secret makes Helm fall back to the last completed one:

kubectl delete secret sh.helm.release.v1.payments-api.v7 -n payments

Then run a plan-equivalent before doing anything else:

helm get manifest payments-api -n payments | kubectl diff -f -

Only delete the pending revision. Deleting the wrong Secret loses that revision's record, and with it the ability to roll back to it. Never delete them all.

"has no deployed releases"

Error: UPGRADE FAILED: "payments-api" has no deployed releases

The first install failed, so there is no deployed revision to upgrade from. Helm will not upgrade a release that never successfully installed.

helm uninstall payments-api -n payments
helm install payments-api ./chart -n payments

Since nothing was ever successfully deployed, uninstalling loses nothing. Check helm list -a first to confirm the status really is failed on revision 1.

"field is immutable"

cannot patch "payments-api" with kind Deployment: Deployment.apps
"payments-api" is invalid: spec.selector: Invalid value: field is immutable

Not a Helm problem. Kubernetes is rejecting the change. Some fields cannot be modified after creation:

ResourceImmutable field
Deployment, StatefulSetspec.selector
Servicespec.clusterIP, and the type in some transitions
PersistentVolumeClaimAlmost everything except size
JobNearly the entire spec

A chart upgrade that changes label conventions hits this constantly, because the selector is derived from labels.

The only fix is replacing the resource:

kubectl delete deployment payments-api -n payments
helm upgrade payments-api ./chart -n payments

That means downtime for that resource. On a StatefulSet, --cascade=orphan deletes the controller while leaving the Pods running, which lets you recreate it without dropping traffic. The Pods are adopted by the new StatefulSet.

CRDs are never upgraded

The quietest of these failures.

Helm installs CRDs from a chart's crds/ directory exactly once, and never upgrades or deletes them. This is deliberate: a botched CRD change can invalidate every custom resource of that type in the cluster.

The consequence is that a chart upgrade can ship a controller expecting a schema the cluster does not have. The symptom is not an upgrade failure at all. The upgrade succeeds, and then applying a custom resource fails validation on a field that should exist.

kubectl get crd <name> -o jsonpath='{.metadata.annotations}'
kubectl apply -f https://.../crds.yaml      # before the chart upgrade

Chart release notes usually mention it. Read them for operator charts specifically.

Preventing all of this

helm upgrade --install payments-api ./chart \
  -n payments \
  -f production.yaml \
  --atomic \
  --timeout 5m

--atomic is the single most valuable flag in Helm. If the release does not become ready within the timeout, Helm rolls it back automatically, so a bad upgrade leaves you at the previous working revision rather than in a half-applied state you have to reason about during an incident.

It implies --wait, so always pair it with --timeout, or a genuinely stuck rollout waits far longer than you want.

Two more habits worth forming:

Pass the full values file every time. helm upgrade discards previously supplied values that are not passed again, so an upgrade with only --set image.tag=x silently loses every other override.

Install the helm-diff plugin and preview before upgrading:

helm diff upgrade payments-api ./chart -f production.yaml

A checklist

  1. helm list -a -n <ns>. The -a reveals failed and pending releases.
  2. helm history <release> -n <ns>. Identify the last deployed revision.
  3. pending-*helm rollback <release> <last-good>.
  4. Rollback also blocked → delete only the pending revision's Secret, then diff.
  5. has no deployed releases → uninstall and install; nothing was ever deployed.
  6. field is immutable → delete the resource and upgrade, or --cascade=orphan for a StatefulSet.
  7. Controller misbehaving after an upgrade → apply the chart's CRDs manually.
  8. Add --atomic --timeout to every upgrade so this stops happening.

Frequently Asked Questions

How do I fix "another operation (install/upgrade/rollback) is in progress"?

The release is stuck in a pending-* state because a previous operation was interrupted. Confirm with helm list -a, which unlike plain helm list shows non-deployed releases, then helm rollback <release> <last-good-revision> to return to the last completed state. If rollback is also blocked, delete the Secret holding the pending revision, sh.helm.release.v1.<release>.v<N> in the release namespace, which makes Helm fall back to the previous one. Confirm nothing is genuinely running first.

What does "has no deployed releases" mean in Helm?

The very first install of that release failed, so there is no successfully deployed revision for Helm to upgrade from. Since nothing was ever running, the fix is to uninstall and install fresh rather than upgrade: helm uninstall <release> then helm install <release> ./chart. Check helm list -a first to confirm the release really is at revision 1 with a failed status, so you are not uninstalling something that did deploy.

Why does Helm say "field is immutable"?

Kubernetes is rejecting the change, not Helm. Certain fields cannot be modified after a resource is created. Most commonly a Deployment or StatefulSet's spec.selector, which a chart upgrade changing its label conventions will alter. The only way through is to delete the resource and let the upgrade recreate it, which means downtime for that workload. For a StatefulSet, kubectl delete statefulset <name> --cascade=orphan removes the controller while leaving the Pods running, so the replacement adopts them without dropping traffic.

Why didn't Helm upgrade my CRDs?

By design. Custom Resource Definitions placed in a chart's crds/ directory are installed once on first install and never touched again on upgrade or uninstall, because an incorrect CRD change can invalidate or destroy every custom resource of that type in the cluster. Helm considers that too destructive to automate. When a chart upgrade includes CRD changes, apply them yourself with kubectl apply -f first. The release notes for operator charts normally say so, and skipping it produces validation errors that look unrelated.

What does --atomic do and should I always use it?

--atomic makes an upgrade all-or-nothing: Helm waits for the release to become ready and, if it does not within the timeout, automatically rolls back to the previous revision. Without it, a failed upgrade leaves the release partially applied with some resources new and some old, which is a bad state to reason about mid-incident. Use it on every upgrade, and always pair it with --timeout, since --atomic implies --wait and will otherwise sit on a genuinely stuck rollout.

Where does Helm store release state, and is it safe to delete?

Helm 3 stores each revision as a Secret named sh.helm.release.v1.<release>.v<N> in the release's own namespace, which is why helm list is namespace-scoped and why helm list -A exists. Deleting a single pending revision's Secret is a legitimate recovery when a release is wedged and rollback is blocked. Deleting others loses your ability to roll back to them, and deleting all of them makes Helm forget the release entirely while its resources keep running.

Learn the underlying concept