kubectl exec, logs, and port-forward
Debug running workloads. Reading logs including crashed containers, getting a shell inside a Pod, ephemeral debug containers, and tunnelling a local port.
Three commands for working with what is already running: read its output, get inside it, and reach it from your machine.
logs
kubectl logs payments-api-7d4f-x8k2
kubectl logs payments-api-7d4f-x8k2 -f # follow
kubectl logs payments-api-7d4f-x8k2 --tail=100
kubectl logs payments-api-7d4f-x8k2 --since=15m
kubectl logs payments-api-7d4f-x8k2 --timestamps
The flag that matters most
kubectl logs payments-api-7d4f-x8k2 --previous
When a container is crash-looping, plain kubectl logs shows the instance that has just started, which during a backoff has produced nothing yet. --previous shows the run that actually failed, and that is where the error is.
If you learn one thing about debugging Pods, this is it. A CrashLoopBackOff with apparently empty logs is nearly always someone forgetting --previous.
Multiple containers
kubectl logs <pod> -c api # one container
kubectl logs <pod> --all-containers # all of them
With more than one container in a Pod, kubectl logs refuses to guess and errors until you pick one.
Across Pods
kubectl logs -l app=payments-api --tail=50 --prefix
kubectl logs -l app=payments-api -f --max-log-requests=10
--prefix labels each line with its Pod name, which is essential when reading three replicas at once. Following many Pods is capped by --max-log-requests (default 5).
For serious multi-Pod log reading, stern remains better than anything built in.
What logs are and are not
kubectl logs reads files on the node written by the container runtime from the container's stdout and stderr. Two consequences:
- An application logging to a file inside the container produces nothing. Log to stdout.
- Logs are deleted with the Pod.
--previousreaches back exactly one container generation, not further. Anything you need to keep requires a log shipper.
exec
kubectl exec -it payments-api-7d4f-x8k2 -- sh
kubectl exec -it payments-api-7d4f-x8k2 -c api -- bash
kubectl exec payments-api-7d4f-x8k2 -- env # single command, no TTY
kubectl exec payments-api-7d4f-x8k2 -- ls /config
The -- separator matters. Everything after it goes to the container, so without it flags like -l are interpreted by kubectl instead.
Use sh rather than bash by default. Most slim images have no bash.
Things worth checking from inside a container:
kubectl exec <pod> -- env | sort # what config did it actually get?
kubectl exec <pod> -- cat /config/app.yaml # is the ConfigMap mounted?
kubectl exec <pod> -- wget -qO- localhost:8080/healthz # does the probe endpoint work?
kubectl exec <pod> -- nslookup payments-db # does DNS resolve?
That first one settles a great many arguments. Checking what the container actually received, rather than what the manifest says, is usually faster than reasoning about the manifest.
Distroless images have no shell
kubectl exec needs a shell inside the container. Distroless and scratch images deliberately have none:
OCI runtime exec failed: exec failed: unable to start container process:
exec: "sh": executable file not found in $PATH
The answer is an ephemeral debug container. A temporary container added to the running Pod, sharing its network namespace:
kubectl debug -it payments-api-7d4f-x8k2 \
--image=nicolaka/netshoot \
--target=api
--target=api shares the target container's process namespace so you can see its processes. nicolaka/netshoot carries curl, dig, tcpdump, netstat and most of what you would want.
The Pod is unchanged when you exit. Nothing is rebuilt, nothing restarts.
To debug a Pod that will not start at all, copy it with a different command:
kubectl debug payments-api-7d4f-x8k2 -it --copy-to=debug-pod --container=api -- sh
And for a node:
kubectl debug node/node-1 -it --image=nicolaka/netshoot
port-forward
kubectl port-forward pod/payments-api-7d4f-x8k2 8080:8080
kubectl port-forward service/payments-api 8080:80
kubectl port-forward deployment/payments-api 8080:8080
Tunnels a local port through the API server to something in the cluster. This is the right way to reach an internal service from a laptop, no Ingress, no NodePort, no exposing anything.
# Reach a database locally
kubectl port-forward -n payments service/postgres 5432:5432
psql -h localhost -U payments
Forwarding to a Service picks one Pod and stays with it; it is not load balanced. Forwarding to a Deployment picks one of its Pods.
Notes worth knowing: 0.0.0.0:8080:8080 as the address binds all interfaces rather than just localhost, the connection drops when the Pod is replaced, and it only handles TCP.
Copying files
kubectl cp payments-api-7d4f-x8k2:/var/log/app.log ./app.log
kubectl cp ./fixture.json payments-api-7d4f-x8k2:/tmp/fixture.json
kubectl cp <pod>:/path ./local -c api # pick a container
kubectl cp requires tar inside the container, so it fails on distroless images just as exec does.
A debugging order that works
kubectl get pods. What state is it in?kubectl describe pod <name>. Do the events explain it?kubectl logs <name> --previous. What did the failing run say?kubectl exec <name> -- env. Did it get the configuration you expected?kubectl debug, if there is no shell, or you need network tools.
Most problems are answered by steps 2 and 3.
Quick reference
| Task | Command |
|---|---|
| Logs | kubectl logs <pod> |
| Crashed container logs | kubectl logs <pod> --previous |
| Follow | kubectl logs <pod> -f |
| One container | kubectl logs <pod> -c <container> |
| Across Pods | kubectl logs -l app=<label> --prefix --tail=50 |
| Shell | kubectl exec -it <pod> -- sh |
| One command | kubectl exec <pod> -- env |
| Debug a distroless Pod | kubectl debug -it <pod> --image=nicolaka/netshoot --target=<c> |
| Debug a node | kubectl debug node/<node> -it --image=nicolaka/netshoot |
| Local access | kubectl port-forward service/<n> 8080:80 |
| Copy a file out | kubectl cp <pod>:/path/file ./file |
Key takeaways
kubectl logs --previousis the crash-debugging command. Plainlogsshows the container that just started.- Only stdout and stderr are captured. An application logging to a file produces nothing.
- Logs die with the Pod, and
--previousreaches back exactly one generation. --separates kubectl's flags from the container's command.kubectl exec <pod> -- envanswers most "why is it misconfigured" questions immediately.- Distroless images have no shell: use
kubectl debugwith an ephemeral container. port-forwardis how you reach an internal Service without exposing it.
Check your understanding
Empty logs on a crash loop
A Pod is in CrashLoopBackOff. kubectl logs returns nothing at all. What is the explanation?
No shell in the container
kubectl exec into a distroless production container fails with "exec: sh: executable file not found in $PATH". What is the right approach?
Frequently Asked Questions
Why does kubectl logs show nothing for a crashing Pod?
Because it reads the container that is running now, and during a CrashLoopBackOff that instance has only just started or has not started at all. The output you need belongs to the previous, terminated instance: kubectl logs <pod> --previous. Pair it with kubectl describe pod, whose events often name the cause outright, and note that --previous only reaches back one container generation. Earlier runs are gone unless you ship logs off the node.
How do I get a shell inside a running Pod?
kubectl exec -it <pod> -- sh, adding -c <container> when the Pod has more than one. Use sh rather than bash, since most slim images do not include bash. The -- separator is required so that everything after it is passed to the container instead of being interpreted by kubectl. For a Pod with several containers, kubectl will not guess and returns an error until you name one.
How do I debug a container with no shell?
Use kubectl debug -it <pod> --image=nicolaka/netshoot --target=<container>. This attaches an ephemeral container to the running Pod that shares its network namespace, and with --target its process namespace, giving you a full toolkit against the live workload without modifying the image or restarting anything. It is the standard answer for distroless and scratch images, and the Pod returns to normal when you exit.
What does kubectl port-forward do?
It opens a tunnel from a local port, through the API server, to a port on a Pod or Service in the cluster, so kubectl port-forward service/payments-api 8080:80 makes the Service reachable at localhost:8080. This is the correct way to reach an internal service or database from a laptop without creating an Ingress or changing the Service type. Forwarding to a Service selects a single Pod and is not load balanced, and the connection drops when that Pod is replaced.
Why don't my application's logs appear in kubectl logs?
Because the application writes to a file inside the container rather than to stdout or stderr. The container runtime only captures those two streams, and that is all kubectl logs can read. The fix is to configure the application to log to stdout, ideally as structured JSON, which also makes logs collectable by a shipper such as Promtail or Fluent Bit. Writing to files inside containers means losing them entirely when the Pod is removed.
Can I view logs from all Pods in a Deployment at once?
Yes, with a label selector: kubectl logs -l app=payments-api --tail=50 --prefix. The --prefix flag labels each line with its Pod name, which is necessary to make sense of interleaved output. Following multiple Pods is capped by --max-log-requests, five by default. For sustained multi-Pod log watching, stern handles it considerably better, with colouring per Pod and automatic pickup of new Pods as they appear.