Monitoring performance¶

Practical 5.7 monitoring-performance-annex.pdf · PDF

Your laptop is on. Nothing has crashed. No error window, no red dot. And yet every click takes two seconds and the fan sounds like a hair dryer. What is wrong?

In The case of the failed service the problem left a body: an exit code, a log line. A slow machine leaves nothing. No log ever says "I am slow". You have to measure. That is what a system administrator does when a colleague at Tramuntana Apps says "my machine is so slow" — and it is what you will do in Practical 5.7.

Slow is not broken: the four resources¶

Remember the kitchen from Unit 1: the CPU is the chef, RAM is the countertop, the disk is the pantry. Now picture Saturday night. The restaurant is full and dishes come out late. Before you shout at the chef, look at where things pile up. There are only four places:

The bottleneck is the slowest resource: it sets the pace of the whole system.

Unit 1 used this word for the road between CPU and RAM. Same idea here: hiring a second chef does nothing if the dishes are waiting at the pantry door. So the rule of the day is: measure first, buy later.

Load average: how many are waiting?¶

$ uptime
 20:54:41 up  1:52,  0 users,  load average: 4.73, 4.17, 2.59
$ nproc
12

The load average is the average number of processes that were running or waiting for a CPU over the last 1, 5 and 15 minutes.

nproc prints the number of cores. One load number alone means nothing: you always compare it with the cores. On that 12-core machine a load of 4.73 is a quiet afternoon. On your 2-vCPU deb-lab it would be a rush:

Read the three numbers as a trend:

• 6.0, 2.1, 0.8 — the last minute is the worst: it started a minute ago.
• 0.8, 2.1, 6.0 — the last 15 minutes were the worst: it is over.

Memory: why "free" lies¶

               total        used        free      shared  buff/cache   available
Mem:            30Gi        13Gi       460Mi       3.4Gi        19Gi        17Gi
Swap:           31Gi        12Ki        31Gi

That is free -h on a 31 GB machine (your 4 GB deb-lab shows about 3.8Gi total). Look: only 460 MiB free, but 17 GiB available. Is the machine out of memory? No.

Linux fills unused RAM with copies of files it has read from disk — the cache (buff/cache) — because empty RAM is wasted RAM. The moment a program needs the space, the cache gives it back.

Available is the honest number: how much memory programs could still get, counting the cache that can be given back.

In the kitchen: a countertop covered with ingredients you took out "just in case" is not a full countertop. You can clear it in a second.

Swap is a reserved area of the disk where the system moves memory that has not been used recently, when RAM really runs out.

Swap keeps the machine alive. But the disk is thousands of times slower than RAM (Unit 1), so a machine that swaps all the time feels frozen.

vmstat: one line, many clues¶

vmstat 1 5 prints one line per second, five times. Here is a calm line and a busy line from the same machine:

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 2  0  11432 654344    928 20571848    0    0     0  3016 11144 27805  8  5 87  0  0  0
 6  0  78892 596348    928 20872744   16 7020 36912 76628 43531 48246 16 15 65  3  0  0

Read the second line: r jumped to 6, bo went from 3,016 to 76,628 and so is 7,020. Something is writing hard to the disk and memory is being pushed to swap. One line of numbers, three clues.

The baseline: what healthy looks like¶

A baseline is a set of measurements of the machine when nothing is wrong.

Without it, numbers under load mean nothing: is wa 4 a lot? Only your baseline can say. Close the browser in the VM, wait one minute, and measure:

sudo apt install sysstat stress-ng   # iostat lives in sysstat; stress-ng creates load
nproc            # how many cores: 2 on deb-lab
uptime           # three numbers well below 2
free -h          # copy total, used, free, buff/cache, available
vmstat 1 5       # copy the LAST line
df -h /          # Use% well under 100
ip -s link       # RX/TX errors and dropped near 0 (the interface that is not lo)

On a resting deb-lab (2 vCPU, 4 GB, GNOME) expect: r 0–1, b 0, si/so 0, id above 90, wa 0–1. Your network interface has a name like enp1s0, not eth0.

Finding the bottleneck¶

In the Exercise sheet of Practical 5.7, three machines at Tramuntana Apps are "slow". A small script, mystery.sh, recreates each complaint for three minutes (bash mystery.sh A, then B, then C). You run the load in one terminal and measure in a second one — the first one is busy.

flowchart TB
    S["<b>It is slow</b><br>uptime: load above the cores?"] --> V["<b>vmstat 1 5</b><br>read from line 2"]
    V -->|"r high, id near 0,<br>us near 100"| CPU["<b>CPU</b><br>too few hands"]
    V -->|"si / so not 0,<br>available low in free -h"| MEM["<b>Memory</b><br>countertop full"]
    V -->|"b above 0, wa high,<br>bo huge, iostat busy"| DISK["<b>Disk</b><br>queue at the pantry door"]
    V -->|"all calm"| NET["<b>Network?</b><br>ip -s link: errors, dropped"]
    CPU --> WHO["<b>top</b><br>P sorts by CPU, M by memory:<br>who is doing it?"]
    MEM --> WHO
    DISK --> WHO

What each load looks like on deb-lab, compared with the baseline:

Load C is the 1993 detail live: the load average goes up although the CPU is not working hard.

The Windows side¶

Windows looks at the same four resources with different tools:

• Task Manager (Ctrl+Shift+Esc) → Performance: one graph each for CPU, Memory, Disk and Ethernet. The Processes page, sorted by a column, says who is responsible.
• Resource Monitor (Win+R → resmon, or the link at the bottom of the Performance page): tabs Overview, CPU, Memory, Disk and Network, process by process.
• Get-Counter in PowerShell, for the command line:

Get-Counter '\Processor(_Total)\% Processor Time' -SampleInterval 2 -MaxSamples 3
Get-Counter '\Memory\Available MBytes'

Windows has no load average; CPU % and the graphs are enough for this course.

Try the smallest CPU load there is, in a PowerShell window of win11:

while ($true) { }

It keeps one core at 100 %. On the 2-vCPU win11, the CPU graph settles near 50 %. Task Manager shows the percentage of the whole machine: one busy core out of two. Ctrl+C stops the loop and the graph falls back.

Key takeaways¶

• A slow machine runs out of one of four resources: CPU, memory, disk or network. The slowest one is the bottleneck.
• The load average counts processes running or waiting over 1, 5 and 15 minutes. Compare it with nproc; on Linux it also counts processes waiting for the disk.
• In free -h, trust available, not free: the cache is given back when programs need it. Constant swap is the real sign of short RAM.
• In vmstat, skip line one: r and id speak about the CPU, si/so about memory, b, wa and bo about the disk.
• A baseline of the healthy machine is what makes the numbers under load meaningful. Measure before you buy.
• On Windows: Task Manager, Resource Monitor and Get-Counter show the same four resources as a percentage of the whole machine.

Quick check¶

Next: now that you can read a machine, change it — its name, its clock, its language and its PATH — in Configuring the system by command.