Linux interview questions and answers for DevOps and SRE

Each digit is a three-bit field for one class of user: owner, group, other. Read is 4, write is 2, execute is 1.

644 is owner read and write, group read, other read. That is the normal mode for a file.

755 is owner read, write and execute, group read and execute, other read and execute. That is the normal mode for a directory or a script.

The part that shows understanding: on a directory, the bits mean something different. Read lets you list names, write lets you create and delete entries, and execute lets you traverse into it. A directory with read but not execute gives you the file names and nothing else, which is why 755 rather than 644 on directories is not arbitrary.

ln file.txt hard.txt      # same inode
ln -s file.txt soft.txt   # contains the path "file.txt"
ls -li                    # first column is the inode number

The practical consequence is that deleting the original leaves a hard link fully working and leaves a symlink dangling.

sudo ss -tlnp | grep :8080

ss is the current tool; netstat is deprecated and often not installed. The flags are worth knowing: t for TCP, l for listening, n for numeric so it does not do DNS lookups, p for the owning process, which needs root to see processes you do not own.

sudo lsof -i :8080 gives the same answer with more detail about the open files.

A good follow-up answer notes that a port held by a process in TIME_WAIT has no owner to show, and that inside a container ss sees only that container's network namespace, so a conflict on the host will not appear there.

Two usual causes.

A deleted file still held open by a process. Unlinking removes the name, but the kernel keeps the data until the last file descriptor closes. A log file rotated out from under a running service is the classic case.

sudo lsof +L1

That lists open files with a link count of zero. Restarting the holding process, or truncating through its descriptor, releases the space.

Exhausted inodes rather than bytes. Millions of tiny files fill the inode table while df -h still shows free space.

df -h /var
df -i /var

If df -i shows 100% use, you need to delete files rather than free bytes, and no amount of removing large files will help.

> truncates the target and writes; >> appends.

Streams are numbered: 0 is stdin, 1 is stdout, 2 is stderr. So:

cmd > out.log          # stdout only, truncating
cmd 2> err.log         # stderr only
cmd > all.log 2>&1     # both into one file
cmd &> all.log         # same thing, bash shorthand
cmd 2>&1 | grep error  # send stderr through the pipe too

Order matters in > all.log 2>&1: stdout is pointed at the file first, then stderr is pointed at wherever stdout currently goes. Writing 2>&1 > all.log sends stderr to the terminal and only stdout to the file, which is a real and common mistake.

The interview point: reaching for kill -9 first is a habit worth arguing against. It risks corrupting state, and a process that ignores SIGTERM is a bug to investigate. Note also that a process stuck in uninterruptible sleep, state D, will not die even from SIGKILL, because it is blocked in the kernel waiting on IO.

Work outward from load, and name what each step rules out.

uptime                  # load average over 1, 5, 15 minutes

Load higher than the core count means processes are waiting. Compare the three numbers to see whether it is rising or recovering.

top -b -n1 | head -20

Read the CPU line rather than just the process list:

free -h                 # is it actually memory
iostat -xz 1 3          # await and %util per device
ss -s                   # socket totals, are we out of connections
dmesg -T | tail -30     # OOM kills and hardware errors

The strongest answers say what they expect to find before running each command, and check dmesg for OOM kills early, since "slow" is often a service being repeatedly killed and restarted.

chmod u+s binary     # setuid,  shows as rws------
chmod g+s shared/    # setgid
chmod +t /tmp        # sticky,  shows as drwxrwxrwt

Add the security note: setuid root binaries are a standard privilege escalation route, and auditing for unexpected ones with find / -perm -4000 is a real hardening step.

systemctl status myapp
journalctl -u myapp -n 100 --no-pager

status gives the current state, the exit code and the last few log lines; journalctl gives the history, which is what you need when it is crash-looping and status only shows the newest attempt. Add -b to limit to this boot, or -f to follow.

Then narrow by cause. A non-zero exit code with nothing in the journal usually means the unit file is wrong rather than the application: a bad ExecStart path, a WorkingDirectory that does not exist, or a User that cannot read its own files.

systemd-analyze verify /etc/systemd/system/myapp.service
systemctl cat myapp        # the effective unit, including drop-ins

systemctl cat is the one people miss. A drop-in under /etc/systemd/system/myapp.service.d/ overrides the unit you are reading, and this shows what is actually in effect. And after editing any unit, systemctl daemon-reload is required or systemd keeps using the old one.

Go up the stack, one layer at a time, so each step eliminates something.

ip addr; ip route          # do we have an address and a route
ping -c3 10.0.1.20         # layer 3, if ICMP is allowed
getent hosts api.internal  # is it name resolution
nc -zv api.internal 8080   # is the specific port reachable
curl -sv http://api.internal:8080/healthz

getent hosts is better than nslookup here because it uses the same resolution path the application does, including /etc/hosts and nsswitch, rather than querying DNS directly.

If the port is refused rather than timing out, something is there and actively saying no, which usually means the service is not listening or is bound to 127.0.0.1. A timeout points at a firewall or security group silently dropping packets.

sudo ss -tlnp | grep 8080  # on the far end, is it 0.0.0.0 or 127.0.0.1

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

Be able to explain each stage: awk takes the first field, sort groups identical lines together because uniq only collapses adjacent duplicates, uniq -c counts each group, sort -rn orders by that count numerically and descending, and head takes the top ten.

The mistake interviewers watch for is omitting the first sort, which makes uniq silently undercount.

A stronger answer scales it. On a very large file, sort spills to disk, and doing the counting in awk avoids that entirely:

awk '{c[$1]++} END {for (i in c) print c[i], i}' access.log | sort -rn | head -10

A zombie is a process that has exited but whose parent has not yet read its exit status, so the kernel keeps the process table entry. It shows as state Z and uses no memory or CPU.

A few transient zombies are normal. A growing pile is a bug in the parent, which is not calling wait(). The real risk is exhausting the process table, since each zombie holds a PID.

You cannot kill a zombie, because it is already dead. Killing or restarting the parent clears them, and if the parent dies they are reparented to PID 1, which reaps them.

ps -eo pid,ppid,stat,cmd | awk '$3 ~ /^Z/'

The container connection is worth raising: an application running as PID 1 in a container inherits the duty of reaping orphans, and most application runtimes do not do it. That is exactly what docker run --init and tini are for.

An inode holds a file's metadata: type, permissions, owner and group, size, timestamps, link count, and pointers to the data blocks.

It does not hold the filename. Names live in directory entries, which map a name to an inode number. That single fact explains several behaviours: hard links are two names for one inode, renaming a file does not touch the inode, and you can delete a file you cannot read as long as you have write permission on the directory.

ls -li file.txt     # inode number
stat file.txt       # everything the inode holds
df -i               # inode usage per filesystem

It also explains running out of inodes with free disk space, since the table is sized at filesystem creation on ext4.

Concretely: dependency-ordered parallel startup rather than sequential runlevels, supervision with configurable restart policies, cgroup-based resource limits per unit, socket and timer activation, a structured and indexed journal, and a consistent declarative unit format instead of a few hundred lines of bespoke shell per service.

[Service]
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5
MemoryMax=512M
User=myapp

The cgroup point matters for a senior answer: systemd puts each unit in its own cgroup, so a restart reliably kills every child process, which an init script's PID file approach frequently failed to do.

The honest trade-offs: it is large and does a lot beyond init, which is the substance of the long-running criticism; binary journals need journalctl rather than grep; and the behaviour is genuinely complex, so debugging ordering problems means understanding Wants, Requires, After and BindsTo rather than reading a script top to bottom.

On Linux it is the exponentially-damped average number of processes in runnable or uninterruptible state over one, five and fifteen minutes. The second half is the part people miss: Linux counts processes blocked on IO, which other Unixes do not.

So a load of 8 on an 8-core box can mean fully utilised CPU, or it can mean eight processes all blocked on a failing disk while the CPU sits idle. Load alone does not distinguish them, which is why it is a starting signal and not a diagnosis.

uptime
vmstat 1 5     # r is runnable, b is blocked

Compare load to core count from nproc, and read the three intervals as a trend: 1-minute well above 15-minute means something is starting now.

The senior addition is pressure stall information, which answers the question load average cannot:

cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory

When the kernel cannot reclaim enough memory it scores every process, roughly by memory footprint adjusted by oom_score_adj, and kills the highest scorer to free the most memory for the least number of kills. It is not the process that requested the allocation, and it is not the oldest, which is why the victim is often a database rather than the leaking process.

dmesg -T | grep -i "killed process"
journalctl -k | grep -i oom
cat /proc/<pid>/oom_score

Protect the processes that matter by biasing the score, from -1000 to 1000:

echo -500 > /proc/<pid>/oom_score_adj

Two points that mark experience. In a container, hitting the cgroup memory limit produces an OOM kill of that container's process even though the host has memory free, which is the exit code 137 you see in Kubernetes. And overcommit settings matter: with the default heuristic the kernel allows more allocation than it can back, so the reckoning arrives at first touch rather than at allocation.

The point of the question is breadth, so move steadily and do not skip layers.

1. 1Resolution. The stub resolver consults nsswitch, so /etc/hosts then DNS. A recursive resolver walks root, TLD and authoritative nameservers unless it is cached.
2. 2Routing. The kernel picks a route and a source address, then resolves the next hop's MAC with ARP.
3. 3TCP. SYN, SYN-ACK, ACK.
4. 4TLS. ClientHello with SNI so a shared host knows which certificate to send, then ServerHello, certificate, chain validation against the local trust store including expiry and hostname, key agreement, Finished. TLS 1.3 needs one round trip, not two.
5. 5HTTP. The request is sent, the server responds, and the connection may be reused for keep-alive.

Strong answers note that failures are diagnosable by layer, and that curl -v shows you exactly where it stopped. They also mention that the certificate is validated against the hostname, which is why an IP address in the URL fails even when the host is reachable.

By default a shell script continues after a failing command and exits 0, which means a broken deploy script reports success.

set -e exits on an unhandled non-zero status. set -u errors on an unset variable, which catches the rm -rf "$PREFIX/" typo where PREFIX was never set. set -o pipefail makes a pipeline fail if any stage fails, not just the last, so curl bad-url | jq . stops being a success.

Where it still does not protect you:

• set -e is ignored in a condition, so if cmd; then is fine, which is intended, but it also does not fire inside && chains the way people expect.
• A failure in a command substitution used in an assignment can be swallowed.
• It does nothing for a command that succeeds while doing the wrong thing.

So the fuller habit is set -euo pipefail, quote every expansion, use "${VAR:?message}" where a value is mandatory, add a trap for cleanup, and run shellcheck in CI.

Establish whether it is bytes or inodes first, because the remedy differs:

df -h; df -i

Then descend by size rather than guessing:

sudo du -xh --max-depth=1 / 2>/dev/null | sort -rh | head

-x keeps it on one filesystem, which stops you walking into network mounts and container overlays and wasting minutes.

Then check the three usual culprits: /var/log for logs that are not rotating, /var/lib/docker on a container host, and deleted-but-open files, which du cannot see at all:

sudo lsof +L1

The operational point worth making: on a full production disk, truncating a live log with : > /var/log/app.log is safer than deleting it, because deleting leaves the space held by the writing process and rotation will not recover it until restart.

ulimit -Sn    # soft open file descriptors
ulimit -Hn    # hard

The one that bites in production is nofile. Every socket is a file descriptor, so a busy server hits the default 1024 and starts failing with "too many open files" while looking otherwise healthy. It presents as intermittent connection failures under load, not as a clear resource error.

Raising it in your shell does nothing for a service started by systemd, which is the trap:

[Service]
LimitNOFILE=65535

Check what a running process actually has, rather than what you think you set:

cat /proc/<pid>/limits

grep -q pattern file
echo "exit status: $?"

long_task &
pid=$!
wait "$pid"

Worth adding: $? is overwritten by every command, including echo, so capture it into a variable immediately if you need it twice.

journalctl keeps history, which is the advantage over tailing a file that may have been rotated.

journalctl -u myapp --since "1 hour ago" --no-pager
journalctl -u myapp -p err          # priority error and above
journalctl -u myapp -b -1           # the previous boot
journalctl -u myapp -o json-pretty  # structured fields

-p err is the one that saves time on a noisy service. -o json-pretty shows the fields you can then filter on directly, which is how you find all messages from one PID across restarts.

The gotcha to mention: the journal is only persistent if it is configured to be. With Storage=volatile or no /var/log/journal directory, everything is in memory and disappears on reboot, which is a genuinely unpleasant discovery during an incident.

journalctl --disk-usage

In order of preference.

A targeted sudo rule. Specific commands, no wildcards that allow shell escapes:

deploy ALL=(root) NOPASSWD: /bin/systemctl restart myapp

Be alert that many commands allow escaping to a shell, so granting sudo vim or sudo find is effectively granting root.

Linux capabilities, when the need is a specific kernel privilege rather than a command. Binding a low port is the standard example:

sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/myapp

That replaces the old pattern of starting as root and dropping privileges.

Group membership, where a group already governs the resource. With the caveat worth stating in any interview: adding someone to the docker group is equivalent to giving them root, because they can mount the host filesystem into a privileged container.

systemd unit properties such as AmbientCapabilities, so the service gets the privilege without any human having it.

/etc/hosts is a local static file mapping names to addresses. DNS is the distributed lookup system. /etc/nsswitch.conf is what decides the order in which they are consulted.

hosts: files dns myhostname

files means /etc/hosts is checked first, which is why an entry there overrides DNS entirely and is a common cause of "it resolves differently on that one box".

This is why getent hosts name is the right diagnostic rather than dig name. dig queries DNS directly and bypasses nsswitch, so it can return the correct record while the application still resolves something else from /etc/hosts or a local caching resolver such as systemd-resolved on 127.0.0.53.

This is usually a queue or a limit rather than a resource, and the interesting answers go to the network stack and to contention.

Connection and socket limits first. Accept queue overflows and conntrack table exhaustion both produce exactly this symptom:

ss -s
netstat -s | grep -i -E "listen|retrans"   # overflows and retransmits
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

Retransmits point at packet loss somewhere in the path. A non-zero and growing "times the listen queue of a socket overflowed" means the application is not accepting fast enough.

Then contention that does not show as utilisation: file descriptor limits near the ceiling, a database connection pool fully checked out, garbage collection pauses inside the runtime, DNS resolution timing out on a stale resolver, and steal time on a shared host.

cat /proc/pressure/io
vmstat 1 10     # cs for context switches

The framing worth offering: "fine on averages" often means bad at the tail, so look at p99 rather than means, and check whether slowness correlates with a specific instance, upstream or time pattern before tuning anything.

grep -rn --include='*.py' --exclude-dir='.git' "TODO" .

-r recurses, -n gives line numbers, --include and --exclude-dir narrow the search, which matters both for speed and for not matching your own node_modules.

Useful variations: -i for case-insensitive, -l to list only filenames, -c to count matches, -w for whole words to stop cat matching concatenate, and -A3 -B3 for context lines around each hit.

Worth mentioning ripgrep if it is available, since it respects .gitignore by default and is substantially faster on a large tree:

rg -t py TODO