Backup Monitoring — How to Know When Your Backups Stop Working

The worst time to discover your backups aren't running is when you need to restore from one.

It happens more often than anyone admits. A backup script that ran faithfully for months silently breaks — maybe a disk filled up, a credential expired, a cron entry got overwritten during an update. The backup process exits with no fanfare, and nobody notices until disaster strikes.

Backups fail in ways that don't announce themselves

A web server that goes down produces immediate, visible effects — users complain, error pages appear, monitoring alerts fire. A backup that stops running produces no signal at all. The system keeps working perfectly. Data keeps flowing. Users are happy. The only thing missing is your safety net.

Common ways backups fail silently:

Credential rotation — your backup tool's database credentials or cloud storage API keys expired
Disk space — the backup destination filled up, writes failed, the script exited
Package updates — a system update broke a dependency or changed a path
Cron entry lost — a server reimage, migration, or config management run cleared the schedule
The script "succeeds" but backs up nothing — an empty database dump still creates a file, and the backup tool reports success

Uptime monitoring won't catch any of these. Your server is up, your app is healthy, HTTP returns 200. The backup is the only thing that's broken.

How to monitor backups with a heartbeat

The fix is simple: make your backup script check in with a heartbeat monitor after every successful run. If the check-in stops, you get alerted.

Here's the pattern:

#!/bin/bash
set -euo pipefail

# Run the backup
pg_dump -U myuser mydb | gzip > /backups/mydb-$(date +%Y%m%d).sql.gz

# Verify the backup file is non-empty
BACKUP_FILE="/backups/mydb-$(date +%Y%m%d).sql.gz"
if [ ! -s "$BACKUP_FILE" ]; then
    echo "Backup file is empty, not sending heartbeat"
    exit 1
fi

# Only ping heartbeat if backup actually succeeded
curl -fsS https://poppaping.com/heartbeat/YOUR_TOKEN

The critical detail: the heartbeat only fires if the backup succeeds and produces a non-empty file. If pg_dump fails, set -e aborts the script before the curl runs. If the dump produces an empty file, the size check catches it.

Setting it up

Create a heartbeat monitor with the interval matching your backup schedule (24 hours for nightly, 1 hour for hourly)
Set a grace period — if your backup normally takes 20 minutes but could take an hour on a heavy day, set the grace to 1-2 hours
Add the heartbeat ping to the end of your backup script, after validation
Connect an alert channel — email, Slack, Discord, whatever your team watches

From that point forward, you have positive confirmation every single day that your backup ran and produced output. The first day it doesn't, you know within hours — not weeks.

What to validate before pinging

Sending the heartbeat after the backup command exits isn't enough. The backup command can "succeed" (exit code 0) while producing garbage. Add validation:

Check file size — an empty or suspiciously small backup file means something went wrong
Check file count — if your backup produces multiple files, verify the expected count
Test decompression — gzip -t backup.gz verifies the archive is readable
For database dumps — check that the dump contains expected tables or a minimum row count
For cloud uploads — verify the upload completed by checking the remote file exists

Only send the heartbeat after your validation passes. This turns your heartbeat into a genuine "backup verified" signal, not just a "backup script ran" signal.

Example: monitoring multiple backup targets

If you back up multiple databases or services, create a separate heartbeat for each:

# Database A backup
pg_dump -U user db_a | gzip > /backups/db_a.sql.gz
[ -s /backups/db_a.sql.gz ] && curl -fsS https://poppaping.com/heartbeat/TOKEN_DB_A

# Database B backup
pg_dump -U user db_b | gzip > /backups/db_b.sql.gz
[ -s /backups/db_b.sql.gz ] && curl -fsS https://poppaping.com/heartbeat/TOKEN_DB_B

# File backup
rsync -a /var/www/ /backups/www/ && curl -fsS https://poppaping.com/heartbeat/TOKEN_FILES

Now you have individual visibility into each backup target. If database B's backup fails but A and the file backup succeed, you know exactly what broke.

The math

A single hour of unplanned downtime costs most businesses thousands of dollars. Discovering you have no backup to restore from turns a 1-hour recovery into a multi-day incident — or worse.

Backup monitoring costs $5/month and takes 5 minutes to set up. It's the cheapest insurance you'll ever buy for your data.

PoppaPing heartbeat monitors are available on all paid plans. Set up a heartbeat for every backup you run, and never be surprised by a silent failure again.