The case of the failed service¶

Exercise the-case-of-the-failed-service-annex.pdf · PDF

Friday, 11:20, Tramuntana Apps. Four services are down: tramuntana-menu, tramuntana-backup, tramuntana-status and tramuntana-report. The person who set them up is on holiday. Your phone is ringing. What do you do first?

A beginner starts changing things until something works — and often breaks two more on the way. A technician asks the machine first. You already have the two tools: systemctl status from Services and systemd and journalctl from Logs and journald. Today you learn to use them as a method.

What you will learn

Follow a fixed method to repair a service: state → evidence → one fix → proof
Read the exit information in systemctl status: failed, auto-restart, status=203/EXEC, status=1/FAILURE
Find the line in journalctl -u that names the cause, and tell systemd's messages from the program's own
Fix faults caused by a wrong path, a missing permission, a missing configuration file or a folder the service cannot write in
Why "just restart it" is not a fix
Write down a diagnosis that another technician could check

Think like a detective¶

The detective (we keep it all page)

A good detective does not arrest the first person she sees. She looks at the scene (state), collects the clue that points at someone (evidence), makes one arrest (one fix) and then proves it in court (proof). A fix nobody checked is just a rumour.

The method fits on a sticky note. Use the same four steps for every case:

Step	Command	You write down
1 · State	`systemctl status NAME`	The `Active:` line and the exit `status=`
2 · Evidence	`sudo journalctl -u NAME -n 20`	The one line that names the cause
3 · One fix	The smallest change that removes the cause	The exact command
4 · Proof	`sudo systemctl restart NAME` · `systemctl is-active NAME`	`active`, and a healthy line in the log

flowchart LR
    A["<b>1 · State</b><br>systemctl status<br>the scene"] --> B["<b>2 · Evidence</b><br>journalctl -u<br>the clue"]
    B --> C["<b>3 · One fix</b><br>smallest change<br>the arrest"]
    C --> D["<b>4 · Proof</b><br>restart + is-active<br>+ healthy log line"]
    D -->|"still failing"| B

Careful: restart-and-see is not a method

A service that fails for a reason fails again when you restart it. All you get is the same error and a longer log. Read first, change second.

Step 1 · Read the scene: the state¶

Start with the whole picture:

systemctl --failed

It lists every unit whose state is failed. Then look at each one with systemctl status NAME. Two lines matter most.

The Active: line says what the service is doing:

`Active:` says	It means
`active (running)`	It works (for now)
`failed (Result: exit-code)`	It died and systemd gave up
`activating (auto-restart)`	It died, and systemd will try again in a moment — and again, and again
`inactive (dead)`	It is simply stopped

Careful: auto-restart hides from --failed

A service with Restart=on-failure that keeps dying is not in the failed state: it is being restarted for ever. So systemctl --failed does not list it. Always check every service you were told about, not only the ones in the list.

The exit status= says where it failed. It is a number with a name:

Exit status	Who failed	Meaning
`203/EXEC`	systemd	It could not run the program in `ExecStart=` at all
`217/USER`	systemd	It could not become the user in `User=`
`1/FAILURE`	The program	The program started, then ended with an error
`9/KILL`	Someone	The process was killed with SIGKILL

Step 2 · Collect the clue: the evidence¶

The status shows only the last few log lines — sometimes none at all, if you look between two restarts. journalctl -u shows all of them:

sudo journalctl -u NAME -n 20

Now ask the most useful question of the day: who wrote the line?

A line that starts with systemd[1]: was written by systemd. If the cause is there, the program probably never started.
A line that starts with the program's own name, such as status-board.sh[4219]:, was written by the program. It did start, and is telling you what it did not like.

A worked example¶

Watch the method on the heartbeat service of Practical 5.6. Someone added the line User=maria to its unit — a typo: the user is marta. After daemon-reload and restart:

$ systemctl status heartbeat
   Main PID: 5282 (code=exited, status=217/USER)
$ sudo journalctl -u heartbeat -n 5
(tbeat.sh)[5282]: heartbeat.service: Failed to determine user credentials: No such process
(tbeat.sh)[5282]: heartbeat.service: Failed at step USER spawning /usr/local/bin/heartbeat.sh: No such process
systemd[1]: heartbeat.service: Main process exited, code=exited, status=217/USER

Read it like a detective:

State: 217/USER — systemd could not become that user.
Evidence: "Failed at step USER" — so the user does not exist. Look at User= in the unit (systemctl cat heartbeat): maria.
One fix: correct the typo with sudo nano, then sudo systemctl daemon-reload.
Proof: sudo systemctl restart heartbeat, systemctl is-active heartbeat → active, and a new heartbeat line in the log.

Two things to notice. The cause was above the "exited" line, not in it. And the lines came from systemd, because the program never got to run.

Careful: the last line is not the cause

The last line is usually systemd saying "it exited". That is the symptom. The cause is one or two lines higher — the last words of that line, such as No such file or directory or Permission denied.

Same status, different clue¶

Two services can show exactly the same status=203/EXEC for different reasons. The status code says where it failed; the last words of the journal line say why:

Last words of the line	Look at
`No such file or directory`	The path in `ExecStart=`: does that file exist? `ls /usr/local/bin`
`Permission denied` (on a program)	Its permissions: `ls -l FILE` — is there an `x`?
`Permission denied` (on a data file or folder)	As which user the service runs (`systemctl cat NAME`, the `User=` line) and what the folder allows (`ls -ld FOLDER`)
`cannot read ...` written by the program	The file it names: does it exist? `ls -l FILE`

Step 3 · Make one arrest: the fix¶

Change one thing, the smallest one that removes the cause. Then you know what fixed it. Useful tools:

Tool	For
`systemctl cat NAME`	Reading the unit: `ExecStart=`, `User=`
`ls -l FILE` · `ls -ld FOLDER`	Permissions and owner
`sudo nano FILE`	Editing a unit or a configuration file
`sudo chmod` · `sudo chown` · `sudo setfacl`	Fixing permissions (the permissions sessions of this unit)
`sudo systemctl daemon-reload`	Only after editing a unit file

Careful: Permission denied is not a reason for 777 or root

When a service cannot write in a folder, two "fixes" work and are both wrong. chmod 777 lets everyone write and delete in it. Removing User= makes the service run as root. Somebody wrote User= there on purpose: the principle of least privilege. Find which user needs which permission on which path, and give exactly that.

Careful: sudo echo ... > file does not write as root

$ sudo echo "OFFICE=Palma" > /etc/tramuntana/status.conf
bash: /etc/tramuntana/status.conf: Permission denied

The > is done by your own shell, before sudo even starts — and your shell is not root. Use an editor (sudo nano FILE) or tee, which does the writing itself:

echo "OFFICE=Palma" | sudo tee /etc/tramuntana/status.conf

When you are stuck, become the service's user and try what it tries. If the unit says User=pau:

sudo su - pau
touch /srv/tramuntana/reports/x

If touch says Permission denied, you have reproduced the bug yourself.

Step 4 · Prove it in court: the proof¶

sudo systemctl restart NAME
systemctl is-active NAME
sudo journalctl -u NAME -n 3

active one second after a restart proves little: a broken program may die a moment later. The real proof is a healthy line in the log — the program saying it is doing its job. When all four cases are solved, this prints active four times:

systemctl is-active tramuntana-menu tramuntana-backup tramuntana-status tramuntana-report

Write it down¶

A diagnosis that only lives in your head is useless to the next technician. For each case, write one sentence of this shape:

The state was ..., the evidence was the line ..., so the cause was ..., we fixed it with ..., and the proof was ...

The Exercise sheet has one case file per service with exactly these boxes. Work in pairs and swap who types after each case. And do not open the script that planted the cases: it would tell you the answers and teach you nothing.

Key takeaways¶

Use the same four steps every time: state (systemctl status), evidence (journalctl -u), one fix, proof.
The exit status says where it failed (203/EXEC: systemd could not run it; 217/USER: no such user; 1/FAILURE: the program ran and complained). The log line says why.
Ask who wrote the line: systemd[1] means the program probably never started; the program's own name means it started and complained.
The cause is usually above the last "exited" line. activating (auto-restart) services do not appear in systemctl --failed.
Fix exactly what is wrong: no chmod 777, no running as root. daemon-reload only after editing a unit file.
A fix is proven by is-active and a healthy line in the log.

Quick check¶

1. A service has failed. What do you do first?

Restart it a few times
Reboot the machine
Read journalctl -u for that service
Reinstall the program

Evidence before changes. A service that fails for a reason fails again after a restart or a reboot. The journal tells you the reason.

2. systemctl status shows status=203/EXEC. What does it mean?

systemd could not run the program in ExecStart=
The program ran and returned an error
A user stopped the service
The unit file is not enabled

203/EXEC is systemd's own failure: it never managed to execute the program — wrong path, no execute permission, or a missing #! line. A program that ran and failed gives 1/FAILURE.

3. A log line starts with status-board.sh[4219]: ERROR: cannot read ... What does the beginning tell you?

systemd refused to start it
The kernel killed the process
The unit file has a typo
The program started and wrote the message itself

The line is tagged with the program's own name and PID, not systemd[1]. So the program did start, and is complaining about something it needs — here, a file it cannot read.

4. A service runs with User=pau and cannot write in its folder, which is drwxr-xr-x root root. What is the best fix?

Remove User=pau so that it runs as root
Give pau write access to that folder
Run chmod 777 on the folder
Add pau to the sudo group

Give exactly the permission that is missing, to exactly the user who needs it (chown pau or an ACL for pau). Running as root or opening the folder to everyone also "works", but breaks least privilege.

5. Two services both show status=203/EXEC. How do you tell their causes apart?

By their PID numbers
By the last words of their journal lines
By whether they are enabled
By restarting both and comparing

The status code says where it failed; the journal says why. No such file or directory points at the path in ExecStart=; Permission denied points at the file's x permission.

6. Three of four broken services appear in systemctl --failed. The fourth shows activating (auto-restart). Why is it missing from the list?

It is working correctly
Its unit file was deleted
systemd keeps restarting it, so it never reaches the failed state
systemctl --failed only shows enabled services

A unit with Restart=on-failure that keeps dying is restarted for ever, so its state is activating (auto-restart), not failed. Check every service you were told about, not only the list.

7. You fixed a service and systemctl is-active says active. Pick the two things that make the proof complete.

A new healthy line from the program in journalctl -u
It is still active a little later, not only one second after the restart
The unit file has been enabled with systemctl enable
The machine has been rebooted once

active right after a restart proves little: a broken program can die a moment later. The proof is the program writing its normal line in the log, and staying active. Enabling and rebooting are about boot, not about this fix.

Next: back to permissions, on the other system. Windows has no three triplets at all: every file has an ACL like the one you built on Linux — NTFS permissions.