File transfer¶

Practical 7.4 file-transfer-annex.pdf · PDF

You need to put the whole documentation of the Tramuntana office server on the server: dozens of files, one of them 5 MB. Tomorrow you will change two of them and send it all again. Next week, again. Do you attach everything to an email? The attachment has a size limit, and the email travels as a postcard.

Moving files between machines is one of the oldest jobs of a network, older even than email. Every developer does it daily: uploading a website, sending a build to a server, keeping a backup copy up to date. Today you will learn the three tools that do it safely, and the one old tool you must refuse.

FTP: the protocol that trusted everybody¶

April 1971: Abhay Bhushan, a student at MIT, writes RFC 114, "A File Transfer Protocol". Moving files was one of the first things anybody wanted from a network, months before the first email with an @. The version still called FTP (File Transfer Protocol) today is RFC 959, from October 1985.

It was designed for a network of researchers who trusted each other. It encrypts nothing. Watch what travels when someone logs in to an FTP server (captured with tcpdump, a program that shows the traffic on a network interface):

# tcpdump -i lo -l -A -n "tcp dst port 21" | grep -a -o -E "(USER|PASS) .*"
USER marta
PASS Tramuntana-2027

The user name and the password, readable by anyone on the path: the Wi-Fi, a switch, every router on the way. The files travel the same way. That is why Firefox 90 and Chrome 95 removed FTP support in 2021.

The armoured van: everything inside SSH¶

The analogy for this page: SSH is an armoured van. You already have an encrypted tunnel to the server, the one you use with ssh. Instead of inventing a new protocol, put the files inside that van. Three tools ride in the same van, on the same port 22, with the same keys:

flowchart LR
    subgraph PC ["Your PC"]
        SFTP["<b>sftp</b><br>walk around and pick"]
        SCP["<b>scp</b><br>one line, one copy"]
        RSYNC["<b>rsync</b><br>send only what changed"]
    end
    SFTP --> VAN
    SCP --> VAN
    RSYNC --> VAN
    VAN["<b>SSH tunnel, port 22</b><br>the armoured van:<br>encrypted, same keys"] --> SRV["<b>Server</b><br>ubu-srv or the online VM"]

Two facts for the curious: since OpenSSH 9.0 (April 2022), scp speaks the SFTP protocol inside; and rsync was first announced in June 1996 by Andrew Tridgell and Paul Mackerras, in Australia.

Getting ready¶

In the practical, a variable keeps the address of your server so you do not type it every time. export makes it live only in that terminal:

mkdir -p ~/transfer-lab/docs && cd ~/transfer-lab
echo "# Tramuntana office server" > README.md
for n in 01-overview 02-shares 03-containers; do echo "# $n" > docs/$n.md; done
head -c 5M /dev/urandom > manual.pdf        # a 5 MB file that plays the role of a PDF
export SRV=user@host                        # write YOUR user and host
ssh $SRV hostname                           # if it asks for a password: ssh-copy-id $SRV

When ssh $SRV hostname prints the server's name without asking for a password, your key is in place and the van is ready.

sftp: walk around and move files¶

sftp $SRV
sftp> pwd                              (where am I on the server?)
sftp> lpwd                             (where am I on my PC?)
sftp> mkdir incoming
sftp> cd incoming
sftp> put README.md                    (my PC -> server)
sftp> ls -l
sftp> get /etc/hostname server-name.txt     (server -> my PC)
sftp> bye

pwd answers Remote working directory: /home/marta; lpwd answers Local working directory: /home/marta/transfer-lab. Two rules make sftp easy:

• put and get are named from your side: put sends to the server, get brings from it.
• An l in front means local: lpwd, lls, lcd act on your PC.

scp: one line, one copy¶

scp manual.pdf $SRV:incoming/               # one file, to the server
scp -r docs $SRV:incoming/                  # a whole folder (-r = recursive)
scp $SRV:/etc/os-release .                  # from the server to here (the dot = this folder)
ssh $SRV ls -l incoming incoming/docs

The shape is scp source destination, just like cp. A path with $SRV: in front is on the server, relative to your home there. Each file prints a progress line:

manual.pdf                              100% 5120KB  38.2MB/s   00:00

scp always sends the whole file, every time. Fine once; wasteful when you send the same folder every evening.

rsync: send only what changed¶

rsync -av ~/transfer-lab/ $SRV:incoming/mirror/

-a (archive) copies folders recursively and keeps dates and permissions; -v (verbose) lists what it sends. Now run it three times and read the last lines.

sending incremental file list
created directory incoming/mirror
./
README.md
manual.pdf
os-release
server-name.txt
docs/
docs/01-overview.md
docs/02-shares.md
docs/03-containers.md

sent 5,245,199 bytes  received 198 bytes  10,490,794.00 bytes/sec
total size is 5,243,367  speedup is 1.00

sending incremental file list
sent 249 bytes  received 13 bytes  524.00 bytes/sec
total size is 5,243,367  speedup is 20,012.85

sending incremental file list
docs/02-shares.md
sent 326 bytes  received 42 bytes  736.00 bytes/sec

About 5.2 MB, then 249 bytes, then 326 bytes. rsync compares both sides first and sends only the differences. The speedup is the total size divided by what really travelled. Byte counts change a little from machine to machine, but the shape does not.

It even works inside a file. Add one line at the end of the 5 MB file and ask for statistics:

Literal data: 1,086 bytes               # sent as new data
Matched data: 5,241,808 bytes           # already on the server: not sent

--delete, but dry-run first¶

If you delete a file locally, rsync leaves it on the server unless you add --delete. That option deletes on the destination whatever no longer exists in the source. With the two sides swapped by mistake, it deletes your work. So the habit is: --dry-run first, read the list, then run it for real.

rm docs/03-containers.md
rsync -av --delete --dry-run ~/transfer-lab/ $SRV:incoming/mirror/

The output prints deleting docs/03-containers.md and ends with (DRY RUN): nothing has happened yet. Run the same line without --dry-run to do it.

Prove the copy is identical¶

"The copy finished" is not proof. A checksum (or hash) is a short fingerprint calculated from every byte of a file: one different byte gives a completely different result. Compute it on both sides:

sha256sum manual.pdf
ssh $SRV sha256sum incoming/mirror/manual.pdf

SHA-256 gives 64 hexadecimal characters. The same 64 characters on both sides means identical content.

Which tool?¶

Key takeaways¶

• FTP (RFC 959, 1985) encrypts nothing: user, password and files travel in clear text. Browsers dropped it in 2021.
• SFTP, SCP and rsync all travel inside SSH, on port 22, with the same keys: the armoured van.
• sftp is interactive (put sends, get brings, l means local); scp copies in one line and always sends the whole file.
• rsync compares both sides and sends only what changed; --delete mirrors deletions, so use --dry-run first.
• A checksum such as sha256sum proves the copy is identical. SFTP and FTPS are different protocols.

Quick check¶

That was the last stop: from the bits inside a CPU to documenting, sharing and moving the work of a real office. Look back at the whole journey on the course home.