RAID is not a backup¶

Exercise raid-is-not-a-backup-annex.pdf · PDF

In LVM and RAID you built a perfect mirror. Then you deleted an invoice, and it vanished from both disks at once. Was that bad luck in a lab? Or does it happen to real people, with real work, on real servers?

It happens. In 1998 it nearly cost a film studio a whole movie. This page tells that story, and then gives you a tool to answer one question for any accident: does my data survive this?

The night Pixar deleted Toy Story 2¶

1998, Pixar, California. A whole studio is making Toy Story 2. Every character, every set, every finished shot is a file on one shared server. The film must be in cinemas in November 1999.

One day somebody runs a delete command (rm, the one with no recycle bin) in the wrong place: at the top folder of the film. Oren Jacob, one of the technical directors, watches it happen. Files start to vanish from his screen. First single pieces, then whole characters, then whole sequences. When the command is stopped, about 90 % of the film is gone.

"No panic", they think. "We have backups." The server was backed up regularly. They go to the backups, and the restore does not give the film back. The backups had been failing for some time, and nobody had noticed, because nobody had ever tried to restore from them.

What would you do now?

In a crisis meeting, Galyn Susman, the supervising technical director of the film, remembers something. She had just had a baby and was working from home, so she had a computer there with a copy of the whole film. Nobody had planned it as a backup. She and Jacob drive to her house, wrap the computer in blankets, strap it into her car and drive it back to the studio. The copy is not from that minute, but the film is there.

Pixar had a backup system. What it did not have was one person who had ever tried to get a file back from it.

Four protections, six accidents¶

Your data can be "protected" in very different ways. Here are four that Tramuntana Apps could have for its project server:

And here are six things that really happen. For each cell: does the data survive with only that protection? Y = it survives, N = it is lost, ? = it depends.

The "One disk dies" row assumes the snapshot lives on the disk that died. Ransomware is malicious software that encrypts your files and asks for money to unlock them; it encrypts everything the infected machine can write to, including any disk that is plugged in.

Read the columns:

• RAID 1 has one Y. It answers one question: a dead disk. Nothing else.
• Deletions, ransomware, slow damage. Anything that follows the original at once (a mirror, a synced folder, a disk always connected) also follows the mistake and the attack. And one copy is not enough: you need several moments in time.
• Fire is real. On 10 March 2021 a fire destroyed the SBG2 data centre of the hosting company OVHcloud in Strasbourg and damaged part of the one next to it. Customers whose only backup was in the same place lost both.
• Only the last column is full of Y. It has two things the others lack: another place and history.

What a backup is¶

Put the Pixar story and the table together and you get a definition with three parts:

A backup is a copy from another moment, in another place, that you have already tried to restore.

flowchart LR
    T["<b>Another moment</b><br>survives deletions,<br>ransomware, slow damage"] --> B["<b>BACKUP</b>"]
    P["<b>Another place</b><br>survives fire, theft,<br>a dead server"] --> B
    R["<b>Tested restore</b><br>proves it really<br>holds your data"] --> B

Take one of the three away and you have something else:

A first plan for Tramuntana Apps¶

Tramuntana Apps has 12 people, one server and no backups. A plan answers four questions:

1. What do you copy?
2. Where do the copies live? (more than one place)
3. How often, and how many old copies do you keep?
4. Who checks that a copy can be restored, and when?

And then the hard question: if everything burns on a Thursday at 17:00, how much work is lost?

A good first plan sounds like this: "Every night, a copy of the shared folder to a second disk in the office, keeping 7 days. Every Friday, a copy that Aina takes home. Once a month, Marta restores one file to check. Fire on Thursday at 17:00: the only copy left outside is last Friday's, so we lose almost a week. We should send a copy outside every night."

Notice what it does: more than one place, something that is not always connected, copies from different days, and one named person who tests a restore. The tools come in the next pages. You plan yours with your group on the Exercise sheet.

Key takeaways¶

• RAID survives a dead disk. It does not survive a deletion, ransomware, slow damage, fire or theft.
• A copy that changes together with the original (a mirror, a synced folder, a disk always plugged in) repeats every mistake and every attack.
• A real backup has three parts: another moment, another place, and a tested restore.
• Pixar lost 90 % of Toy Story 2 in 1998: its backups had been failing and nobody had tested a restore. A copy at home, from another day, saved the film.
• A backup nobody has restored is a hope, not a backup.

Quick check¶

That closes Unit 4. Next, Unit 5 is about administering the system: it starts by dusting off the terminal and teaching it new tricks (streams, pipelines, globbing) — Back to the shell.