Skip to content

Services and systemd

Practical 5.6 services-and-systemd-annex.pdf · PDF

Run ps -e on deb-lab right after you log in. Dozens of processes are already there: cron, sshd, cupsd... Nobody typed them. They were running before you logged in, and they will keep running after you log out. Who started them?

In the last session you saw that a process belongs to the terminal that launched it: close the terminal and sleep 300 & dies. So how does a web server stay up for a whole year with nobody logged in? Every website, every chat server and every printer queue you use depends on the answer.

What you will learn

  • Explain what a service (or daemon) is and what systemd does as PID 1
  • Read every line of systemctl status: loaded, enabled, active, main PID, last log lines
  • Start, stop and restart a service — and why start is not enable
  • Write a small unit file and run your own service
  • Show that systemd brings back a service that dies
  • Do the same with Windows services, in services.msc and in PowerShell

What is a service?

A service is a program that runs in the background, with no window and no user in front of it, waiting to do a job.

Unix people call a service a daemon. That is why so many names end in d: sshd (remote logins), cupsd (printing), systemd itself.

The kitchen (we keep it all page)

Remember the kitchen of Unit 1. Not every worker cooks to order. The dishwasher runs all day. The fridge keeps things cold. Someone answers the phone. Nobody orders "wash a plate": those workers are the services.

And someone must open the kitchen every morning, switch them on in the right order, and switch one back on when it trips. That is the shift manager: systemd.

Careful: a service is not a special kind of program

A service is an ordinary process. You can see it with ps and it has a PID like any other. What makes it a service is who starts it and watches it: systemd, not a person at a keyboard.

systemd, the shift manager

systemd is the program the kernel starts first (PID 1). It starts every service, knows which one depends on which, watches them and keeps what they print.

A little history. Until 2010, Linux started its services with a pile of shell scripts run one after another, a system called SysV init. It was slow, and nobody watched a service once it had started. In 2010 Lennart Poettering and Kay Sievers, engineers at Red Hat, presented systemd (Poettering's blog post "Rethinking PID 1" is dated 30 April 2010). Its ideas: start services in parallel, know their dependencies, watch them, keep their output.

It was — and still is — controversial, because it does so much. Debian 8 (April 2015) made it the default after a long public argument. Today it is PID 1 on Debian, Ubuntu, Fedora, Arch and most other distributions. Your deb-lab runs systemd version 257.

flowchart TB
    K["<b>Linux kernel</b><br>boots the machine"] -->|"starts PID 1"| S["<b>systemd</b> (PID 1)<br>the shift manager"]
    subgraph SVC ["Services (no window, no user)"]
        direction LR
        C["<b>cron</b><br>scheduled jobs"]
        SSH["<b>sshd</b><br>remote logins"]
        P["<b>cupsd</b><br>printing"]
    end
    S -->|"starts, watches, restarts"| SVC
    S -->|"starts"| L["Login screen<br>then your shell"]

Careful: systemd is not Linux

Linux is the kernel. systemd is one program that most distributions chose to run as PID 1. Some distributions use a different one.

Units and systemctl

A unit is anything systemd manages, described by a small text file. A service is a unit whose name ends in .service.

One command talks to systemd: systemctl (verb first, then the unit). Look at the scheduler cron:

$ systemctl status cron
● cron.service - Regular background program processing daemon
     Loaded: loaded (/usr/lib/systemd/system/cron.service; enabled; preset: enabled)
     Active: active (running) since Wed 2027-01-27 08:02:11 CET; 3h 20min ago
   Main PID: 612 (cron)
     CGroup: /system.slice/cron.service
             └─612 /usr/sbin/cron -f
Line Tells you
Loaded: + path Where the unit file is, and that systemd has read it
enabled / disabled Will it start by itself at boot?
Active: What it is doing now: active (running), inactive (dead) or failed
Main PID: The process, with the PID you met in the last session
Last lines The service's latest messages (the next session is all about them)

The dot at the start is green when active, white when inactive and red when failed. The status opens in a pager: leave it with q.

Two more useful questions:

systemctl list-units --type=service --state=running   # every running service
systemctl --failed                                    # anything broken?

On a healthy machine, systemctl --failed answers 0 loaded units listed.

Stopping needs an administrator

Try to stop cron as a normal user:

$ systemctl stop cron
Failed to stop cron.service: Access denied

Changing services changes the whole machine, so it needs sudo (on the GNOME desktop a password window appears instead). With sudo systemctl stop cron the status says inactive (dead). Start it again and look: the Main PID is a new number. It is a new process.

Start is not enable

Here is the trap of the day. A service has two separate switches:

Now At every boot
On systemctl start systemctl enable
Off systemctl stop systemctl disable
Ask systemctl is-active systemctl is-enabled
Both at once systemctl enable --now · systemctl disable --now

restart means stop + start. You use it after changing a service's configuration.

$ sudo systemctl disable cron
Removed '/etc/systemd/system/multi-user.target.wants/cron.service'.
$ systemctl is-enabled cron ; systemctl is-active cron
disabled
active

disable only removed a link in a folder that systemd reads at boot. The service is still running. It will not come back after the next reboot. enable creates that link again.

Careful: disable does not stop, enable does not start

Running now and starting at boot are two different switches. If you want both, say so: sudo systemctl enable --now cron.

Anatomy of a unit file

You can read the unit file of any service with systemctl cat:

# /usr/lib/systemd/system/cron.service
[Unit]
Description=Regular background program processing daemon
Documentation=man:cron(8)
After=remote-fs.target nss-user-lookup.target

[Service]
EnvironmentFile=-/etc/default/cron
ExecStart=/usr/sbin/cron -f $EXTRA_OPTS
IgnoreSIGPIPE=false
KillMode=process
Restart=on-failure
SyslogFacility=cron

[Install]
WantedBy=multi-user.target

Three sections. One line in each is worth knowing by heart:

Section Key line Meaning
[Unit] Description= The name shown in status
[Service] ExecStart= The program to run, with its full path
[Service] Restart=on-failure If the process dies badly, start it again
[Install] WantedBy=multi-user.target What enable hooks into: "the system is up, with network, ready for users"

Where unit files live:

  • /usr/lib/systemd/system/ — units that come with installed packages. Do not edit them.
  • /etc/systemd/system/ — your units.

Your own service: the Tramuntana heartbeat

Tramuntana Apps wants every machine to write a heartbeat line every 10 seconds, so the administrator can see it is alive. That is Practical 5.6 — follow the Exercise sheet (the Practical 5.6 button). The idea in four steps:

flowchart LR
    A["<b>1 · Script</b><br>/usr/local/bin/heartbeat.sh<br>chmod +x"] --> B["<b>2 · Unit file</b><br>/etc/systemd/system/<br>heartbeat.service"]
    B -->|"daemon-reload"| C["<b>3 · Start</b><br>systemctl start"]
    C -->|"enable"| D["<b>4 · At every boot</b><br>survives a reboot"]

The program is a plain Bash loop:

#!/bin/bash
while true; do
    echo "heartbeat from $(hostname): load $(cut -d" " -f1 /proc/loadavg)"
    sleep 10
done

The unit that turns it into a service:

[Unit]
Description=Tramuntana heartbeat

[Service]
ExecStart=/usr/local/bin/heartbeat.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start heartbeat
systemctl status heartbeat

daemon-reload tells systemd to read the unit files again. It never notices a new or edited file by itself. The status then shows something like:

● heartbeat.service - Tramuntana heartbeat
     Loaded: loaded (/etc/systemd/system/heartbeat.service; disabled; preset: enabled)
     Active: active (running) since Mon 2026-10-05 20:53:39 UTC; 11s ago
   Main PID: 4088 (heartbeat.sh)
...
Oct 05 20:53:39 deb-lab systemd[1]: Started heartbeat.service - Tramuntana heartbeat.
Oct 05 20:53:39 deb-lab heartbeat.sh[4088]: heartbeat from deb-lab: load 6.30
Oct 05 20:53:49 deb-lab heartbeat.sh[4088]: heartbeat from deb-lab: load 6.35

Look at the bottom: the echo lines of your script are there. You did not redirect anything. systemd catches everything a service prints. Keep that in mind for the next session.

Kill it and watch it come back

Now be cruel: sudo kill -9 on the Main PID. Ask for the status at once:

systemd[1]: heartbeat.service: Main process exited, code=killed, status=9/KILL
systemd[1]: heartbeat.service: Failed with result 'signal'.
systemd[1]: heartbeat.service: Scheduled restart job, restart counter is at 1.
systemd[1]: Started heartbeat.service - Tramuntana heartbeat.

A new Main PID, active (running) again. That is Restart=on-failure at work — the shift manager switching the dishwasher back on. This is the whole reason services exist: a person does not have to be there.

Careful: stop a service through its manager

kill -9 is not how you stop a service: with Restart= it simply comes back. sudo systemctl stop heartbeat does not trigger a restart, because a stop asked through systemd is not a failure.

When it does not start

You see It means
status=203/EXEC systemd could not run the program: wrong path in ExecStart=, no x permission, or the first line #!/bin/bash is missing
The unit file ... changed on disk. Run 'systemctl daemon-reload' You edited the file but systemd still uses the old one
Unit heartbeat.service not found The file has another name or is not in /etc/systemd/system
has a bad unit file setting A typo such as [Services]: systemd ignored the section, so there is no ExecStart=

systemd-analyze verify /etc/systemd/system/heartbeat.service prints nothing when the file is correct, and names the bad line otherwise.

Windows services

Same idea, other names. Press Win+R and type services.msc: a long list with Name, Status (Running or empty) and Startup type. Double-click Print Spooler — the service that queues documents for the printer — to see the Start and Stop buttons.

Linux (systemd) Windows
Look at one systemctl status cron Get-Service Spooler
List running systemctl list-units --type=service --state=running Get-Service, piped to Where-Object Status -eq Running
Start / stop systemctl start / stop Start-Service / Stop-Service
Restart systemctl restart Restart-Service
At boot: yes systemctl enable Set-Service -StartupType Automatic
At boot: no systemctl disable Set-Service -StartupType Manual or Disabled
Graphical tool — services.msc, Task Manager → Services

The Windows startup types:

  • Automatic — starts at boot.
  • Automatic (Delayed Start) — starts a little after boot.
  • Manual — starts only when something asks for it.
  • Disabled — cannot be started.

As on Linux, changing services needs an administrator: open PowerShell with Run as administrator.

Get-Service Spooler
Stop-Service Spooler
Start-Service Spooler
Set-Service Spooler -StartupType Manual

While the spooler is stopped, the list of printers in Settings does not load. Put it back to Automatic when you finish.

Key takeaways

  • A service (daemon) is an ordinary background process started and watched by the system, not by a person.
  • systemd is PID 1 on Debian and most distributions: it starts services, knows their dependencies, restarts them and keeps their output.
  • Everything systemd manages is a unit; systemctl status shows loaded, enabled, active, the main PID and the last log lines.
  • start/stop act now; enable/disable act at every boot. enable --now does both.
  • A unit file has [Unit], [Service] (ExecStart=, Restart=) and [Install]; yours go in /etc/systemd/system, followed by daemon-reload.
  • Windows has the same idea: services.msc, startup types, and Get-Service / Start-Service / Set-Service.

Quick check

1. What is a service (daemon)?
  • A program that runs with administrator rights
  • Any program installed with apt
  • A background process that the system starts and watches
  • A script that runs only once at boot

A service is an ordinary process with no window and no user. What makes it a service is that systemd (on Windows, the service manager) starts it, watches it and can restart it.

2. A service is running and you run sudo systemctl disable on it. What happens?
  • It keeps running, but will not start at the next boot
  • It stops now and never starts again
  • It stops now and starts again at the next boot
  • Nothing happens until you reboot twice

disable only removes the boot link. Running now is the other switch: use stop, or disable --now for both at once.

3. You changed a line in /etc/systemd/system/heartbeat.service. What must you run before restarting the service?
  • systemctl enable heartbeat
  • chmod +x on the unit file
  • systemctl daemon-reload
  • systemctl is-active heartbeat

systemd reads unit files only when told to. daemon-reload makes it read them again; then restart applies the change. Without it you get the warning "changed on disk".

4. You kill -9 the main process of your service and a second later it runs again with a new PID. Why?
  • kill -9 does not work on services
  • The kernel restarts every killed process
  • The unit file has Restart=on-failure
  • The service is enabled, so it restarts

Restart=on-failure tells systemd to start the service again when it dies badly — and being killed by a signal counts. enable only matters at boot.

5. Where should you put the unit file of a service you write yourself?
  • In /etc/systemd/system
  • In /usr/lib/systemd/system
  • In /usr/local/bin
  • In your home folder

/etc/systemd/system is for the administrator's own units. /usr/lib/systemd/system belongs to installed packages, and /usr/local/bin is where the script itself goes.

6. Your new service shows status=203/EXEC. Which two could be the cause?
  • The path in ExecStart= has a typo
  • The script has no execute permission
  • The service is not enabled
  • The service printed too many lines

203/EXEC means systemd could not run the program at all: the file is not where ExecStart= says, or it cannot be executed (no x, or no #!/bin/bash line). Being disabled only affects boot.

7. On a Windows PC, the print spooler must not start at boot but should start when something needs it. Which command?
  • Stop-Service Spooler
  • Set-Service Spooler -StartupType Manual
  • Set-Service Spooler -StartupType Disabled
  • Get-Service Spooler

The Manual startup type means "only when something asks for it". Disabled would stop it from starting at all, and Stop-Service acts only now — the same now / at-boot split as on Linux.

Next: your heartbeat has been writing all night. Where did those lines go, and how do you find one? — Logs and journald.