Kubernetes Monitoring

Kubernetes ships almost no monitoring. kubectl top does not work on a fresh cluster, there is no metric history, and container logs vanish when a Pod is deleted.

Everything below has to be installed deliberately.

metrics-server is not a monitoring system

metrics-server collects current CPU and memory usage from each kubelet and serves it through the Metrics API. It is what kubectl top and the Horizontal Pod Autoscaler read.

kubectl top nodes
kubectl top pods --containers

NAME                     CPU(cores)   MEMORY(bytes)
payments-api-7d4f-x8k2   143m         310Mi
payments-api-7d4f-m3p9   156m         298Mi

It stores nothing. There is no history, no querying, no alerting, just the last reading. If kubectl top fails or your HPA shows <unknown>, metrics-server is the first thing to check:

kubectl get deployment metrics-server -n kube-system

Install it, but do not mistake it for observability.

Prometheus is the actual answer

The near-universal choice, and the practical way to install it is the kube-prometheus-stack chart, which bundles Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics with dashboards and alert rules already wired together:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

Assembling those components individually is a genuine time sink for no benefit.

Three sources feed it, and knowing which produces what saves a lot of confusion:

kube-state-metrics is the one people miss. It does not measure resource usage; it exposes what the API server knows. That is where kube_deployment_status_replicas_unavailable, kube_pod_container_status_restarts_total and kube_pod_status_phase come from, and those are the metrics most of your alerts should be built on.

Scraping your own applications

Expose Prometheus metrics from your service, then tell Prometheus to collect them with a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payments-api
  namespace: payments
  labels:
    release: monitoring        # must match the Prometheus instance's selector
spec:
  selector:
    matchLabels:
      app: payments-api
  endpoints:
    - port: metrics            # the NAME of a Service port, not a number
      interval: 30s

Two things break this more often than anything else. The port must be the name of a port on the Service, so the Service must declare named ports. And the ServiceMonitor's own labels must match what the Prometheus resource selects, with kube-prometheus-stack that is usually release: <helm release name>.

# Is it being scraped?
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090
# then open localhost:9090/targets

What to actually watch

For applications, the RED signals: Rate, Errors, Duration. For resources, the USE signals: Utilisation, Saturation, Errors.

Cluster-level things worth a dashboard:

A useful query. Pods restarting repeatedly in the last hour:

increase(kube_pod_container_status_restarts_total[1h]) > 3

And CPU throttling, which almost nobody watches and which explains a lot of unexplained latency:

rate(container_cpu_cfs_throttled_periods_total[5m])
  / rate(container_cpu_cfs_periods_total[5m]) > 0.25

Alerts worth having

Alert on symptoms users feel, not on every deviation. An alert nobody acts on trains people to ignore the channel.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payments-api
  namespace: payments
  labels:
    release: monitoring
spec:
  groups:
    - name: payments-api
      rules:
        - alert: PodCrashLooping
          expr: increase(kube_pod_container_status_restarts_total{namespace="payments"}[15m]) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.pod }} is crash-looping"

        - alert: DeploymentUnavailable
          expr: kube_deployment_status_replicas_unavailable{namespace="payments"} > 0
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "{{ $labels.deployment }} has unavailable replicas"

The for clause matters as much as the expression. Without it, every transient blip pages someone.

A short, honest starting set: crash-looping Pods, a Deployment with unavailable replicas, nodes not ready, disk above 85%, certificates expiring within two weeks, and your own error rate and latency SLOs.

Logs

Container logs live on the node and are deleted with the Pod. kubectl logs reads them directly, which is fine for debugging and useless afterwards:

kubectl logs <pod>
kubectl logs <pod> --previous        # the run that crashed
kubectl logs -l app=payments-api --tail=100 --prefix
kubectl logs <pod> --since=1h

For anything you need to keep, run a log shipper as a DaemonSet. Loki with Promtail or Alloy is the common pairing with Prometheus and Grafana and is far cheaper to operate than Elasticsearch; the ELK or OpenSearch stack remains the choice where full-text search over large volumes matters.

Log to stdout in JSON. Writing to files inside the container means the collector cannot see them, and structured logs make fields queryable instead of regex-matched.

Events expire

Events are how Kubernetes explains itself, and they are only retained for one hour by default:

kubectl get events -A --sort-by=.lastTimestamp
kubectl get events --field-selector involvedObject.name=payments-api-7d4f-x8k2

That hour is routinely shorter than the time between an incident and someone investigating it. Ship events into your logging system, kube-prometheus-stack does not do this by itself, and the gap is easy to miss until you need an event that has already expired.

Tracing

Metrics say something is slow; traces say where. OpenTelemetry is the standard instrumentation layer, with Tempo, Jaeger or a commercial backend behind it.

Worth adding once you have several services calling each other. Before that, metrics and logs cover most of what you need.

Quick reference

---

Key takeaways

• Kubernetes monitors nothing out of the box. metrics-server, Prometheus and log shipping are all installed by you.
• metrics-server has no history. It exists to serve kubectl top and the HPA.
• Install kube-prometheus-stack rather than assembling the components yourself.
• kube-state-metrics supplies object state, which is where most useful alerts come from.
• ServiceMonitors fail on two things: the endpoint port must be a port name, and the labels must match Prometheus's selector.
• Watch CPU throttling. It causes latency that is invisible from inside the application.
• Events expire after an hour: ship them somewhere before you need them.
• Alert on symptoms, and always set for.

---

Check your understanding

---

Frequently Asked Questions

Why does kubectl top not work on my cluster?

metrics-server is not installed, or is not healthy. It is a separate component that collects current CPU and memory from each kubelet and serves the Metrics API that kubectl top reads, and most distributions do not ship it by default. Check with kubectl get deployment metrics-server -n kube-system. The same component backs the HorizontalPodAutoscaler, so an HPA showing <unknown> targets usually has the same root cause. The other possibility there being Pods with no resource requests.

What is the difference between metrics-server and Prometheus?

metrics-server serves only current CPU and memory readings, keeps no history, and exists to support kubectl top and autoscaling. Prometheus is a time-series database that scrapes many sources on an interval, stores the results, and supports querying, dashboards and alerting. They are complementary rather than alternatives: you generally run metrics-server for the HPA and Prometheus for everything a human looks at.

What is kube-state-metrics and do I need it?

It exposes the state of Kubernetes objects as Prometheus metrics. How many replicas a Deployment wants versus has available, which phase each Pod is in, container restart counts, job completions. It measures the API server's view rather than actual resource consumption, which is why it complements the kubelet's cAdvisor metrics rather than duplicating them. Most genuinely useful alerts are built on it, so yes. It is included in kube-prometheus-stack.

How do I keep logs after a Pod is deleted?

You cannot with kubectl logs. Container logs live on the node and are removed with the Pod, and --previous only reaches back one container generation. Run a log shipper as a DaemonSet on every node: Promtail or Grafana Alloy feeding Loki is the usual pairing alongside Prometheus, while Fluent Bit into OpenSearch or a commercial backend suits environments needing heavier full-text search. Have applications log to stdout in JSON, since files written inside the container are invisible to the collector.

Why do my Kubernetes events disappear?

Events have a one-hour retention by default, controlled by the API server's --event-ttl. That is often shorter than the gap between an incident and someone investigating it, so the scheduling failure or probe error that explains everything has already expired. Ship events into your logging system with a tool such as the Kubernetes Event Exporter. This is not included in kube-prometheus-stack and is easy to overlook until the moment you need an event that is gone.

What should I alert on?

Symptoms that users experience, not every metric that moves. A short, honest starting set: Pods crash-looping, Deployments with unavailable replicas, nodes not Ready, disk above 85%, certificates expiring within two weeks, and your own error rate and latency against an agreed SLO. Always include a for clause so transient blips do not page anyone. Alerts nobody acts on are worse than no alerts, because they teach people to ignore the channel that also carries the real ones.