Compression, encryption, integrity¶

Practical 4.7 compression-encryption-integrity-annex.pdf · PDF

You send a folder of photos to a friend as a .zip. Someone says: "It's zipped, so it's safe." Is it? Smaller, yes. But can a stranger open it? And how would your friend know nobody changed a photo on the way?

Those are three different questions, and they need three different tools. In restic and the restore drill restic answered all three at once, without asking. Tramuntana Apps must send its weekly archive to the accountant: it should be small, nobody else may read it, and she must know it arrived unchanged. Today you do each job by hand.

Three jobs, one parcel¶

Think of sending a parcel by post:

flowchart LR
    F["<b>Archive</b><br>the jumper"] -->|"compress"| C["<b>Smaller</b><br>vacuum-packed"]
    C -->|"encrypt"| E["<b>Secret</b><br>box with a padlock"]
    E -->|"sha256sum"| S["<b>Fingerprint</b><br>the weight on the label"]
    S -->|"sent with the parcel"| R["<b>Accountant</b><br>checks the weight,<br>opens the padlock,<br>unpacks"]

Three things to keep straight:

• Archive is not compress. tar only packs many files into one. The compressor makes that one file smaller. zip does both at once, file by file, and Windows opens it without installing anything.
• Old and new. gzip is from 1992 (Jean-loup Gailly and Mark Adler) and is everywhere. zstd (Zstandard) is from 2016 (Yann Collet, at Facebook): the same job, much faster. You will measure it.
• All of this is lossless: what comes out is identical, byte for byte, to what went in. JPEG and MP3 belong to another family, lossy compression: they throw information away for ever. Not today.

Build a sample and race four compressors¶

Install the tools and build a 50 MB sample with three kinds of content:

sudo apt install zstd zip unzip age
mkdir -p ~/lab37/sample
cd ~/lab37
seq 1 4000000 > sample/numbers.txt
cp -r /usr/share/common-licenses sample/licences
head -c 20M /dev/urandom > sample/random.bin
tar -cf sample.tar sample
ls -lh

seq writes the numbers from 1 to 4,000,000: 30 MB of text. The licences are a few ordinary texts. random.bin is 20 MB of random bytes. tar -cf packs the folder into one file without compressing: sample.tar is about 50M.

Now the race. time measures how long a command takes (read the real line); -k keeps the original file:

time gzip -k sample.tar
time zstd -k sample.tar
time xz -k sample.tar
time zip -q -r sample.zip sample
ls -l

Results from a test VM with 2 vCPU (your sizes will be almost the same; your times depend on your PC):

ls -l gives bytes: 29,589,180 bytes is 29.6 MB. Notice that nobody gets under 20 MB. Why?

Why random data does not shrink¶

Compress the two big files of the sample on their own:

gzip -k sample/numbers.txt sample/random.bin
ls -l sample
rm sample/*.gz

A compressor works by finding repetition and describing it more briefly: "the digits 1 to 9 again, with a new prefix". Numbers in order are full of patterns. Random bytes have no pattern to describe, so there is nothing to shorten, and the compressor's own bookkeeping makes the file slightly bigger.

Did compression lose anything? Unpack the .zst into another folder and compare:

mkdir out
tar -xf sample.tar.zst -C out
diff -r sample out/sample

diff prints nothing: every byte came back.

A secret: encryption with age¶

The office Wi-Fi password should not travel as plain text. age is a small, modern encryption tool. With -p it encrypts with a passphrase (a password, usually long). Type one and do not leave it empty: if you just press Enter, age invents a passphrase and prints it.

cp ~/tramuntana-share/admin/wifi.txt .
cat wifi.txt
age -p -o wifi.txt.age wifi.txt
file wifi.txt.age

file says age encrypted file. Now decrypt it (-d) and compare; then try once with a wrong passphrase:

age -d -o wifi-back.txt wifi.txt.age
diff wifi.txt wifi-back.txt
age -d wifi.txt.age

diff is silent: the decrypted file is identical. With the wrong passphrase you get age: error: incorrect passphrase, and nothing opens.

Unchanged? Checksums with sha256sum¶

A checksum is a fingerprint of a file: a short string computed from every byte. Change one bit and you get a completely different fingerprint. SHA-256 is the algorithm; sha256sum is the command. You used it before to check a downloaded ISO.

Write the fingerprints of the two files you would send into a list, then check them (-c):

sha256sum sample.tar.zst wifi.txt.age > SHA256SUMS
cat SHA256SUMS
sha256sum -c SHA256SUMS

Each fingerprint is 64 hexadecimal characters (256 bits ÷ 4 bits per hex digit). The check answers sample.tar.zst: OK and wifi.txt.age: OK. Now somebody adds one byte on the way (>> adds to the end of a file):

echo x >> wifi.txt.age
sha256sum -c SHA256SUMS

sample.tar.zst: OK
wifi.txt.age: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

Choosing¶

When you finish the practical, clean up: cd ~ and rm -r ~/lab37. Every step, with space for your numbers, is on the Practical 4.7 sheet.

Key takeaways¶

• Three different jobs: compression makes it smaller, encryption makes it secret, a checksum proves it is unchanged.
• tar only packs; gzip, zstd, xz and zip compress. zstd is the fast modern choice, zip the one Windows opens.
• Compression is lossless here and works by describing repetition: text shrinks, random data (and anything already compressed or encrypted) does not.
• age -p encrypts with a passphrase; a wrong passphrase opens nothing. A zip password is weak.
• sha256sum -c checks a list of fingerprints: one changed byte gives FAILED.
• Order matters: compress first, then encrypt.

Quick check¶

This week you typed the same backup commands again and again. Next you write them once, in a file, and the computer types them for you: Shell scripts.