Linux Performance Monitoring and Troubleshooting
A method for Linux performance triage. CPU, memory and I/O in order, what load average really counts, why free memory looks low, and reading iowait correctly.
Why Performance Troubleshooting Needs a Method
When a Linux host feels slow, the real bottleneck might be CPU, memory, disk I/O, or network behavior. Guessing based on one symptom usually wastes time.
A better approach is the USE method:
- Utilization: how busy is the resource?
- Saturation: is work piling up and waiting?
- Errors: are failures or drops occurring?
CPU Analysis
top and htop
top
htop
These tools show running processes, CPU usage, memory usage, and load averages. htop is friendlier when installed, but top is usually available by default.
Load averages with uptime
uptime
This shows 1, 5, and 15 minute load averages. A high load number can mean many tasks are running or waiting, not only that CPUs are maxed out.
CPU details with lscpu
lscpu
Use this to learn how many CPUs and cores the system has.
Per-CPU stats with mpstat
mpstat -P ALL 1 5
This is useful when one core is overloaded while others are mostly idle.
Tip: High load average does not automatically mean CPU exhaustion. Disk waits can also raise load.
Memory Analysis
Linux uses memory aggressively for cache, so low raw free memory is not always a problem. What matters is whether the system is under pressure.
free -h
free -h
Look at total, used, free, and especially available memory.
vmstat
vmstat 1 5
vmstat shows process activity, memory, swap, I/O, and CPU state over time. Ongoing swap in or swap out activity can suggest memory stress.
/proc/meminfo
grep -E 'MemTotal|MemFree|MemAvailable|SwapTotal|SwapFree' /proc/meminfo
This provides detailed kernel memory counters and is useful for scripts or deeper inspection.
Note: A system with low free memory but healthy available memory and little swapping may still be fine.
Disk and I/O Analysis
Storage bottlenecks can make the whole server feel slow even when CPU usage looks reasonable.
iostat
iostat -xz 1 5
This helps you inspect device utilization, throughput, and wait times. High utilization plus long waits often points to disk pressure.
iotop
sudo iotop
Use this to find which processes are generating the most disk I/O in real time.
Capacity checks with df and du
df -h
du -sh /var/log/*
df -h shows filesystem usage, while du shows which directories consume space. Full filesystems can break applications, databases, and package managers.
Network Analysis
Network congestion or connection storms can also create performance symptoms.
Socket summaries with ss -s
ss -s
This provides a quick summary of socket states and can help reveal abnormal connection counts.
Interface counters from /proc/net/dev
cat /proc/net/dev
This file shows bytes, packets, drops, and errors for each interface.
Traffic viewers
sudo iftop
sudo nethogs
iftop shows bandwidth by connection. nethogs attributes traffic to processes. They are especially useful when you know the network is busy but not why.
Putting It Together During an Incident
Suppose users report that an API is suddenly slow. A practical first pass could be:
uptime
top
free -h
vmstat 1 5
iostat -xz 1 5
ss -s
You might conclude:
- very busy CPUs point to compute saturation
- high I/O wait points to storage delays
- swap activity points to memory pressure
- unusually high socket counts point to traffic spikes or connection leaks
This evidence helps you narrow the bottleneck before making changes.
Common Mistakes
Avoid these habits:
- treating load average as the same thing as CPU percent
- checking only one metric and stopping there
- restarting services before collecting enough data
- failing to compare against a normal baseline
Performance issues often span multiple subsystems, so keep gathering evidence before acting.
What You Should Remember
Break Linux performance work into CPU, memory, disk, and network analysis. Use top, uptime, mpstat, and lscpu for CPU; free -h, vmstat, and /proc/meminfo for memory; iostat, iotop, df, and du for storage; and ss -s, /proc/net/dev, iftop, and nethogs for networking.
Most importantly, use a method such as USE so you troubleshoot from evidence rather than instinct.
Test Your Understanding
Exercise 1: Interpreting load average
Why can a high Linux load average appear even when CPU is not fully busy?
Exercise 2: Understanding free memory
Which memory value is often more useful than raw free memory for a quick Linux health check?
Exercise 3: USE method thinking
In the USE method, what does the S stand for?
Frequently Asked Questions
How do I tell whether a server is CPU, memory or I/O bound?
Work through four commands in order. uptime gives load averages. Compare them against your core count, not to each other. top or htop shows whether CPU time is user, system or wa (I/O wait); high wa with low user means the disk is the constraint. free -h shows memory, reading the available column rather than free. iostat -xz 2 shows per-device utilisation and await. High load with idle CPUs almost always means I/O or lock contention, not compute.
What does load average actually measure?
The average number of processes either running or waiting to run over the last 1, 5 and 15 minutes, and on Linux, unlike some Unixes, it also counts processes blocked on disk I/O. It must be read relative to core count: 4.0 saturates a 4-core box and is light on 16. The three numbers matter as a trend, since a 1-minute figure far above the 15-minute one means the problem is arriving now. nproc gives your core count.
Why does free -h show almost no free memory?
Because Linux uses otherwise idle memory as page cache, and that is the correct behaviour. Unused RAM is wasted RAM. The free column is genuinely unused memory and will always look small on a healthy server; the number to read is available, which accounts for cache the kernel can reclaim instantly. Real memory pressure shows up as swap activity in vmstat 1 (the si and so columns) or as OOM-killer messages in dmesg, not as a low free figure.
How do I find what is using all the CPU or memory?
ps aux --sort=-%cpu | head and ps aux --sort=-%mem | head give an immediate ranked answer, which is often quicker than reading top. In htop, F6 sorts interactively. For a process you have identified, pidstat -p <pid> 1 shows its usage over time, and for memory specifically pmap -x <pid> or smem distinguishes shared from private memory. Worth doing before concluding a process is large, since ps counts shared libraries against every process using them.
What is iowait and what do I do about it?
wa in top is the percentage of time CPUs sat idle with at least one outstanding disk request, so it means the CPU had nothing to do but wait. Confirm with iostat -xz 2: %util near 100 and a rising await identify the saturated device, and iotop -o names the processes responsible. The fixes are structural rather than tunable: faster storage, fewer synchronous writes, more caching, or moving the workload. Note that iowait can fall when CPU load rises, because the CPUs are then busy, so never read it alone.
Which performance tools should I install before I need them?
htop for an interactive overview, iotop for per-process disk activity, sysstat for iostat, pidstat and sar, and ncdu for disk usage. sar is the one worth setting up in advance, because it records history, without it, a question like "what did memory do at 03:00 last night" has no answer. Installing these during an incident wastes the time when you least have it, so add them to your base image or configuration management.