Linux Performance Monitoring and Troubleshooting

Why Performance Troubleshooting Needs a Method

When a Linux host feels slow, the real bottleneck might be CPU, memory, disk I/O, or network behavior. Guessing based on one symptom usually wastes time.

A better approach is the USE method:

• Utilization: how busy is the resource?
• Saturation: is work piling up and waiting?
• Errors: are failures or drops occurring?

CPU Analysis

top and htop

top
htop

These tools show running processes, CPU usage, memory usage, and load averages. htop is friendlier when installed, but top is usually available by default.

Load averages with uptime

uptime

This shows 1, 5, and 15 minute load averages. A high load number can mean many tasks are running or waiting, not only that CPUs are maxed out.

CPU details with lscpu

lscpu

Use this to learn how many CPUs and cores the system has.

Per-CPU stats with mpstat

mpstat -P ALL 1 5

This is useful when one core is overloaded while others are mostly idle.

Tip: High load average does not automatically mean CPU exhaustion. Disk waits can also raise load.

Memory Analysis

Linux uses memory aggressively for cache, so low raw free memory is not always a problem. What matters is whether the system is under pressure.

free -h

free -h

Look at total, used, free, and especially available memory.

vmstat

vmstat 1 5

vmstat shows process activity, memory, swap, I/O, and CPU state over time. Ongoing swap in or swap out activity can suggest memory stress.

/proc/meminfo

grep -E 'MemTotal|MemFree|MemAvailable|SwapTotal|SwapFree' /proc/meminfo

This provides detailed kernel memory counters and is useful for scripts or deeper inspection.

Note: A system with low free memory but healthy available memory and little swapping may still be fine.

Disk and I/O Analysis

Storage bottlenecks can make the whole server feel slow even when CPU usage looks reasonable.

iostat

iostat -xz 1 5

This helps you inspect device utilization, throughput, and wait times. High utilization plus long waits often points to disk pressure.

iotop

sudo iotop

Use this to find which processes are generating the most disk I/O in real time.

Capacity checks with df and du

df -h
du -sh /var/log/*

df -h shows filesystem usage, while du shows which directories consume space. Full filesystems can break applications, databases, and package managers.

Network Analysis

Network congestion or connection storms can also create performance symptoms.

Socket summaries with ss -s

ss -s

This provides a quick summary of socket states and can help reveal abnormal connection counts.

Interface counters from /proc/net/dev

cat /proc/net/dev

This file shows bytes, packets, drops, and errors for each interface.

Traffic viewers

sudo iftop
sudo nethogs

iftop shows bandwidth by connection. nethogs attributes traffic to processes. They are especially useful when you know the network is busy but not why.

Putting It Together During an Incident

Suppose users report that an API is suddenly slow. A practical first pass could be:

uptime
top
free -h
vmstat 1 5
iostat -xz 1 5
ss -s

You might conclude:

• very busy CPUs point to compute saturation
• high I/O wait points to storage delays
• swap activity points to memory pressure
• unusually high socket counts point to traffic spikes or connection leaks

This evidence helps you narrow the bottleneck before making changes.

Common Mistakes

Avoid these habits:

• treating load average as the same thing as CPU percent
• checking only one metric and stopping there
• restarting services before collecting enough data
• failing to compare against a normal baseline

Performance issues often span multiple subsystems, so keep gathering evidence before acting.

What You Should Remember

Break Linux performance work into CPU, memory, disk, and network analysis. Use top, uptime, mpstat, and lscpu for CPU; free -h, vmstat, and /proc/meminfo for memory; iostat, iotop, df, and du for storage; and ss -s, /proc/net/dev, iftop, and nethogs for networking.

Most importantly, use a method such as USE so you troubleshoot from evidence rather than instinct.

Test Your Understanding