restic and the restore drill¶

Exercise restic-and-restore-drill-annex.pdf · PDF

Scroll back through the photos on your phone. You can jump to any day of last year in one second, and the phone never asked you to keep "a full album plus every change since". Wouldn't it be nice if your backups worked like that?

In Backups you had to choose. The rsync mirror was fresh but had no past. The tar archive had the past but was old. The ideal tool would run every night, stay small, keep old versions, and restore any of them easily. That tool exists, and it is free: restic.

What you will learn

What a repository and a snapshot are in restic, and why ten snapshots do not take ten times the space
Create an encrypted repository, back up a folder several times and compare two snapshots
Restore one folder, or everything, and prove the result with diff -rq
Check the health of a repository with restic check
Run a restore drill against the clock and measure it with RPO and RTO
Why forgetting the repository password means losing every backup

A photo album of your folder¶

restic is a free backup program, started by Alexander Neumann and maintained by volunteers. Three words carry the whole idea:

Word	Meaning
Repository	The place where restic keeps everything: a folder (today on `/mnt/backup`) or a server far away. It is encrypted with a password
Snapshot	The state of your folder at one moment: "Thursday 12:30". You can go back to any snapshot
Deduplication	Before saving a piece of a file, restic checks whether the repository already has that exact piece. If yes, it only writes a note: "same as before"

Picture the repository as a photo album of your folder. Every night you add a full-page photo of everything. But the album is clever: where tonight's photo is identical to an older one, it glues nothing new and writes "see page 1".

Every page looks complete, like a full backup.
The album grows only by what changed, like an incremental.
At restore time there is no chain to follow: you open one page.

flowchart LR
    subgraph REPO["Repository /mnt/backup/restic (the album)"]
        S1["<b>Snapshot 1</b><br>page 1: every piece<br>stored for real"]
        S2["<b>Snapshot 2</b><br>page 2: one new piece,<br>the rest 'see page 1'"]
        S3["<b>Snapshot 3</b><br>page 3: two new pieces,<br>the rest 'see page 1'"]
    end
    S2 -.->|"same pieces"| S1
    S3 -.->|"same pieces"| S1

The lock¶

The repository is encrypted with AES-256: whoever steals the backup disk reads nothing. The price is printed on the screen when you create it:

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

Two ways to lose an encrypted backup

The disk, and the password. Nobody can reset it, not even an administrator: the password is the key of the encryption. In a company it lives in a password manager, and two people know where.

Repository and first snapshot¶

Mount the backup disk from Backups if it is not mounted (after a reboot it is not), and install restic:

findmnt /mnt/backup
sudo mount LABEL=backup /mnt/backup
sudo apt install restic
restic version

findmnt prints nothing when the disk is not mounted; LABEL=backup finds it by the label you gave it with mkfs. Now create the repository. Today's password is tramuntana. restic asks for it at every command and shows nothing while you type: that is normal.

restic -r /mnt/backup/restic init
ls /mnt/backup/restic
export RESTIC_REPOSITORY=/mnt/backup/restic

ls shows six names: config data index keys locks snapshots. The export line tells this terminal where the repository is, so you do not type -r ... every time. It lives only in that terminal: open a new one and you get Fatal: Please specify repository location until you run it again.

Take the first snapshot:

restic backup ~/tramuntana-share
restic snapshots
du -sh /mnt/backup/restic

Files:          32 new,     0 changed,     0 unmodified
...
processed 32 files, 23.002 MiB in ...
snapshot 3f2a9c1e saved

The 8 characters after snapshot are its ID (yours will be different). restic snapshots lists one line.

You cannot open your files inside the repository

Look inside /mnt/backup/restic/data: file names that are long numbers, contents encrypted. Only restic, with the password, can turn them back into files.

A working day, a second snapshot¶

Change the folder the way a working day would, and back it up again:

cd ~/tramuntana-share
echo 'v3' > design/logos/icon.svg
echo 'Hotel Ponent' > admin/client-2.txt
rm dev/tapes-app/README.md
cd ~
restic backup ~/tramuntana-share
restic snapshots

This time: Files: 1 new, 1 changed, 30 unmodified, and Added to the repository is only a few KiB. Compare the two snapshots with their IDs:

restic diff FIRST-ID SECOND-ID
du -sh /mnt/backup/restic

+    .../admin/client-2.txt
M    .../design/logos/icon.svg
-    .../dev/tapes-app/README.md

+ is new, M is modified, - is removed. And du -sh says about 1.1M, before and after: two snapshots of 23 MiB in about one megabyte. Two reasons: deduplication, and compression (the fake video is all zeros).

Restore, and prove it¶

Two ways to restore:

restic restore latest --target / --include ~/tramuntana-share/admin
restic restore latest --target ~/rtest

latest is the newest snapshot (an ID works too). --target is where to write.
The first line puts one folder back in its place: restic keeps the full path, so target / means "where it was".
The second line writes the whole snapshot into an empty folder, to look at it without touching the live data. The copy appears in ~/rtest/home/USER/tramuntana-share.

Prove the second one:

diff -rq ~/rtest$HOME/tramuntana-share \
  ~/tramuntana-share
restic check

$HOME is filled in by the shell with your home folder. diff -rq prints nothing: no difference. restic check ends with no errors were found.

Would you notice one wrong byte?¶

A dying disk or a bad cable changes one byte of the backup. Would you know? restic stores a checksum of every piece, and restic check --read-data reads every piece again and compares. On a repository damaged on purpose it answers ciphertext verification failed and then Fatal: repository contains errors. Better to learn that today than on the day you need the backup.

Careful: written is not readable

"The backup finished without errors" means it was written without errors. Only restic check and a test restore tell you it can be read.

The restore drill¶

A fire drill is boring, and it is why people get out of a burning building calmly. A restore drill is the same for data. You work in pairs, A and B, each at your own PC:

Both take a fresh snapshot: restic backup ~/tramuntana-share. From now on the backup disk is off limits.
Swap seats. You have two minutes to do two quiet damages in your partner's ~/tramuntana-share (delete a folder, change one word inside a file, rename a file...) and write them down, hidden.

Back to your own seat. Start the clock. Restore and find what was done:

rm -r ~/rtest
restic restore latest --target ~/rtest
diff -rq ~/rtest$HOME/tramuntana-share \
  ~/tramuntana-share

Repair: make the live folder identical to the restored one, with the tool from Backups:

rsync -av --delete \
  ~/rtest$HOME/tramuntana-share/ \
  ~/tramuntana-share/

Run the diff -rq again. Stop the clock when it prints nothing. Your partner reveals the damages: did you find both?

diff finds the damage you would never spot by eye, like one changed word inside a file. That is why you compare with a tool. The worksheet for the drill is on the Exercise sheet.

Restoring is not the easy part

It is the part you do at the worst moment, nervous, with the boss behind you. Whoever has done it once does it in five minutes. Whoever has not reads the manual for the first time that day.

RPO and RTO: two numbers for every plan¶

The drill gave you two numbers. They have names:

	Question	You decide it with
RPO (recovery point objective)	How much work may we lose? (time since the last good backup)	How often you back up
RTO (recovery time objective)	How long may we be stopped? (time until we work again)	How fast you can restore, and how often you rehearse

Example: backups run every night at 23:00. The disk dies on Thursday at 17:00, and the restore takes 2 hours. You lose everything done since Wednesday 23:00 (Thursday's work), and people work again at 19:00.

The same jobs on Windows¶

Same ideas, other names. File History (Control Panel → System and Security → File History) copies the user's folders (Documents, Pictures, Desktop...) to another drive on a schedule and keeps old versions. To get one back: right-click a file → Restore previous versions.

Job	Linux	Windows
Copy a folder so that both sides are equal	`rsync -a --delete`	`robocopy SOURCE DEST /MIR`
Keep old versions of my files on another drive	`restic`	File History
The same tool on both	`restic`	`restic` (it also runs on Windows)

/MIR is --delete

robocopy /MIR deletes in the copy what is gone in the original, exactly like rsync --delete. A mirror, on any system, is not a history.

Key takeaways¶

A restic repository holds every snapshot: the state of your folders at one moment. Each snapshot restores alone.
Deduplication stores each identical piece once, so ten snapshots cost little more than one.
The repository is encrypted: lose the password and the backups are lost for ever.
restic restore latest --target ... brings files back; diff -rq proves the restore; restic check proves the repository can be read.
A restore drill rehearses recovery under pressure. RPO = how much work you may lose; RTO = how long you may be stopped.
On Windows: File History keeps old versions; robocopy /MIR mirrors, like rsync --delete.

Quick check¶

1. In restic, what is a snapshot?

The state of the backed-up folders at one moment
A copy of the whole disk with the operating system
A list of the files that were deleted that day
The password that unlocks the repository

A snapshot is one page of the album: the folder as it was at that moment. You can restore any snapshot on its own, without a chain of older ones.

2. Two snapshots of a 23 MiB folder, and the repository takes about 1 MB. Why?

Identical pieces are stored once, and the data is compressed
restic only saves the file names, not their content
The second snapshot replaced the first one in the repository
restic keeps the files on the original disk, not in the repository

Deduplication means a piece already in the repository is not stored again, only noted. On top of that the data is compressed, and the fake video of zeros shrinks to almost nothing.

3. You lose the password of a restic repository. What now?

Nobody can read those backups any more
The system administrator can reset it
restic sends a reset link by e-mail
The backups open without it after 30 days

The password is the key of the encryption. Without it the repository is noise, for you and for any thief. That is why it lives in a password manager that two people know about.

4. What does restic restore latest --target ~/rtest do?

Writes the files of the newest snapshot into ~/rtest
Makes a new snapshot of the folder ~/rtest
Deletes the newest snapshot from the repository
Compares the newest snapshot with ~/rtest

latest picks the newest snapshot and --target says where to write it. restic keeps the full path, so the files land in ~/rtest/home/USER/....

5. Why do you restore into an empty folder and then run diff -rq?

To prove the backup really holds your data
To make the repository smaller on the disk
To delete the snapshots that are too old
To encrypt the restored files once again

A backup is only proven by a restore. Restoring into an empty folder does not touch the live data, and diff -rq lists every difference. Silence means identical.

6. Backups run every night at 23:00. The disk dies on Thursday at 17:00 and the restore takes 2 hours. What happens?

Thursday's work is lost and people work again at 19:00
Nothing is lost and people work again at 17:00
The whole week is lost and people work again on Friday
Two hours are lost and people work again at 23:00

The last good backup is Wednesday 23:00, so everything after it is lost: that gap is the RPO. The 2 hours of restore are the RTO: 17:00 + 2 h = 19:00.

7. On Windows, which tool keeps old versions of your files on another drive?

File History
robocopy /MIR
Task Manager
Disk Management

File History copies the user's folders to another drive on a schedule and keeps old versions (Restore previous versions). robocopy /MIR is a mirror: like rsync --delete, it has no past.

restic compressed, encrypted and checked your data without asking. Next you do each of those three jobs by hand, to know exactly what they are: Compression, encryption, integrity.