What Breaks When You Back Up a Live Site

The first time I ran our new backup script against a Drupal site we maintain, the VM ran out of disk. So I cleared some space and ran it again. It ran out of disk again. This went on longer than I'd like to admit: run the script, watch it die partway through, delete the giant half-written archive it left behind to claw back enough room to try again, tweak something, run it again. The script wasn't wrong, exactly. It was doing precisely what I'd told it to, which was to tar up everything, including the one folder that had no business being in there.

A Backup Script Is Easy to Write and Easy to Get Wrong

Backing up a live site is not the same job as copying a folder, and the dangerous part is that the broken version still looks like it's working. The disk-space thing was just the failure loud enough to notice. The quieter ones, a backup that silently skips half your data, or exits with an error every night until the team learns to ignore the alert, don't announce themselves at all. You find out on the morning you go to restore.

Everything below is something we got wrong before we got it right, mostly on that same script. I didn't sit down knowing the five things that would bite; I found each one by tripping over it, and the version we trust now is basically the scar tissue from fixing them one at a time. The site is Drupal on an nginx and PHP-FPM stack, but none of this is Drupal-specific. If you're archiving a live web root with tar, it applies.

1. Tar Walks Into Everything, Including the 50GB You Didn't Mean to Copy

What I'd missed was scope: tar follows your web root into every directory under it, and on a real site the big ones are exactly where you don't want it to go. This Drupal site had the usual culprit, a user-uploads folder (Drupal's sites/default/files) that had quietly grown to dozens of gigabytes, plus symlinks pointing at large external stores for images, video, worksheets, and generated documents. My script said "tar up the web root," so tar dutifully walked into all of it. Each run tried to compress tens of gigs onto a VM that didn't have tens of gigs free, filled the disk, died, and left a huge partial archive sitting there eating the space I needed for the next attempt. Hence the delete-and-retry loop.

It took me a few rounds of that to stop blaming the disk and look at what I was actually asking tar to copy. Once the files folder was the obvious suspect, the fix was to be explicit about exclusions and to treat uploads as a separate job from code. We exclude the big uploads path and every symlinked store from the main site archive, so the code backup stays small and fast, then back up the uploads folder on its own schedule in its own mode. Code changes constantly and is tiny; uploads are huge and barely change. Lumping them into one archive on one schedule was the mistake underneath all the others.

There's a trap hiding inside the fix, too, and it cost me another round. A symlink and its real target are two different paths. The uploads folder was reachable by two names, the symlink and the real directory, so excluding just the one I'd thought of still let all the data back in through the other. The exclude list has to name the real path, not whichever name you reached for first.

2. Exit Code 1 Doesn't Mean Your Backup Failed

Once the disk stopped filling up, the next surprise was that the script reported failure on a run that had worked fine. tar routinely exits with code 1 on a live site, and if you treat that as failure your backups will "fail" every single night while they're actually healthy. The reason is mundane: files change while tar is reading them. A log rotates, a cache file gets rewritten, a session file vanishes, all mid-archive. When that happens tar prints "file changed as we read it" and exits 1.

Exit 1 from tar means "some files differed or vanished as I read them." Exit 2 and above mean a real, fatal error. A script that checks if exit code != 0 treats the first, totally normal case as a catastrophe. So you get one of two bad outcomes: either the script aborts a perfectly good backup, or someone wires up alerting, it cries wolf nightly, and the whole team learns to ignore backup alerts, which is far more dangerous.

The fix is to treat exit codes 0 and 1 as success and only fail on 2 or higher. We also pass tar the flags that quiet the expected noise (--warning=no-file-changed, --warning=no-file-removed) so the log stays readable and a real warning still stands out. In practice the check is this small:

tar -czf "$ARCHIVE" --warning=no-file-changed "$@"
status=${PIPESTATUS[0]}

# exit 1 = "files changed/vanished as read", normal on a live site.
# exit 2+ = a real failure.
if [ "$status" -le 1 ] && [ -f "$ARCHIVE" ]; then
  echo "backup ok"
else
  echo "backup failed (tar exit $status)" >&2
fi

"The backup ran and some files were in flux" is the normal state of a live site, not an incident.

3. An Exclude You Didn't Verify Is a Guess

Here's the part that follows directly from the symlink mistake above: an exclude flag you set but never checked is just a hope. Exclude patterns are fussy, a leading slash, a trailing wildcard, a path relative to the wrong base, and the pattern silently matches nothing. tar won't tell you your exclude did nothing. It just quietly includes the thing you meant to skip, and you find out when the archive is three times the size you expected, assuming you're even watching the size. Which, the first few times, I wasn't.

So we verify instead of assuming. After the archive is built, the script lists its contents and confirms the uploads path genuinely isn't in there, and logs the result either way:

# List the archive and count any entries under the path we excluded.
# wc -l always exits 0, so this never trips an exit-code check
# the way `grep -c` (which exits 1 on zero matches) would.
count=$(tar -tzf "$ARCHIVE" | grep "uploads/" | wc -l)

if [ "$count" -eq 0 ]; then
  echo "verified: uploads/ excluded"
else
  echo "WARNING: $count uploads/ entries leaked into the archive" >&2
fi

It's a few seconds of work that converts "I think the exclude is right" into "the archive provably does not contain that directory." Any backup exclusion you care about is worth a check that it actually took effect, because a backup is exactly the thing you don't get to test by accident.

4. Two Backups at Once Is Worse Than None

This one I headed off rather than suffered, because I'd seen it elsewhere. You assume a nightly job finishes before the next night fires, and for a while it does. Then the site grows, the archive takes longer, one slow night overruns the next trigger, and now two tar processes are writing at once, fighting over I/O and disk, each slowing the other down, with a real chance of a truncated or interleaved archive at the end. A backup that corrupts itself is arguably worse than no backup, because you might trust it.

What you want is a lock that guarantees only one run at a time. We use flock on a lockfile: the script grabs an exclusive lock when it starts, and a second run that finds the lock held exits immediately instead of piling on. The operating system releases the lock automatically when the process ends, even if it crashes, so there's no stale-lock cleanup to get wrong. One job runs; the overlapping one steps aside and tries next time.

5. Your Backup Archive Is a Pile of Secrets

A full-system backup is one of the most sensitive files you own, and the default permissions on a new file are nowhere near careful enough for it. Think about what's inside a real backup: server config, TLS private keys, SSH keys, the system password and shadow files, Let's Encrypt certificates, application secrets. Rolled into one tarball, that archive is enough to impersonate the whole server. Dropped in a backup directory with ordinary permissions, any user on the box can read it.

We lock it down at two levels. The backup directory itself is restricted so only root can enter it, and each archive is written as owner-read/write only (chmod 600) the moment it's created. This matters more than it looks, because backups travel: they get copied offsite, pulled down to a laptop to test a restore, synced to object storage. You want the file to be useless to anyone who shouldn't have it no matter where it ends up, not just safe in the directory you first wrote it to.

One More: Never Delete Old Backups Without Being Sure

Pruning old backups to reclaim space is reasonable. Doing it silently is how you find out your retention window was too short only after the files are already gone. We kept automatic cleanup but made deletion deliberate: run the script by hand and it lists exactly which old archives it would remove and asks first; run it from cron, where nobody's watching, and it deletes nothing unless you explicitly pass a flag that says you meant it. Given how much trouble the delete-and-retry loop already caused me, I wasn't about to let the cleanup step delete something I actually wanted.

The Theme Underneath All Five

Every one of these came from the same place the hard-to-find bugs come from: the gap between a command that works on your laptop and one that holds up against a live system. My script worked perfectly against a test folder. It fell apart the moment it met a real site, with a multi-gigabyte uploads folder, symlinks to huge external stores, files changing underfoot, cron timing that drifts, and an archive full of secrets. The idle-folder version passed every time, which is exactly why it taught me nothing until it ran somewhere real.

And none of the fixes are complicated. Exclude what's big and back it up separately, accept that live files move, check that your excludes actually took, lock against overlap, treat the archive as the sensitive thing it is. I just didn't know I needed any of them until a VM kept running out of disk and I spent an afternoon deleting failed archives before the obvious finally landed. That afternoon is the whole reason the current script is any good.

Want Infrastructure That's Actually Been Pressure-Tested?

Backups, deploys, and the quiet plumbing that keeps a site running are easy to set up and easy to get subtly wrong, and you usually find out at the worst possible moment. We've spent 25 years running production systems and cleaning up after the ones that weren't set up carefully, whether we built them or inherited them. If you're not certain your backups would actually restore, that's a good thing to find out on a calm day rather than a bad one.

Start a conversation about your infrastructure, or see how we help teams through our consulting and development services.