Linux Tutorial / Build Server Backups and Independent Health Checks
Reliable operation needs both recoverable backups and checks that reveal failures. Validate archive creation, integrity, restoration, HTTP behavior, and storage separately, and provide an external way to detect failure or nonexecution.
Connect the procedure proven in manual backup and restore verification to scheduled execution. Verify backup creation, independent restoration, and failure reporting separately.
System configuration exercises use a conventional Ubuntu 26.04 LTS or Rocky Linux 9 VM as the reference environment. Containers, WSL, and other releases can differ in service and package configuration. Prepare console recovery and record your versions using the virtual machine setup lesson.
Define the data and recovery scope
The example handles ordinary static files with controlled changes, such as lesson 33 content. Back up service configuration, certificates, DNS information, and deployment procedures separately; use application-consistent tools for databases. The local archive illustrates the workflow and is not independent protection against disk failure or account compromise.
Choose readable source files and a private writable backup directory. Do not schedule everything as root simply to bypass read permissions. Production backups need a deliberate account, approved paths, key management, and retention policy.
Create a backup script with explicit source and destination
mkdir -p "$HOME/bin"
cat > "$HOME/bin/course-backup.sh" <<'EOF'
#!/usr/bin/env bash
set -u
set -o pipefail
umask 077
if (( $# != 2 )); then
printf "Usage: %s SOURCE_DIRECTORY BACKUP_DIRECTORY\n" "$0" >&2
exit 2
fi
source_dir=$(realpath -e -- "$1") || exit 1
backup_dir=$(realpath -e -- "$2") || exit 1
[[ -d "$source_dir" && -d "$backup_dir" ]] || exit 1
case "$backup_dir/" in
"$source_dir/"*) printf "Backup must be outside source\n" >&2; exit 1 ;;
esac
work=$(mktemp -d "$backup_dir/backup.XXXXXXXX") || exit 1
if ! tar -czf "$work/data.tar.gz" -C "$source_dir" .; then
printf "Archive failed: %s\n" "$work" >&2; exit 1
fi
if ! tar -tzf "$work/data.tar.gz" > /dev/null; then
printf "Archive listing failed\n" >&2; exit 1
fi
if ! (cd "$work" && sha256sum data.tar.gz > data.tar.gz.sha256); then
printf "Checksum creation failed\n" >&2; exit 1
fi
if ! date --iso-8601=seconds > "$work/created-at.txt"; then exit 1; fi
printf "complete\n" > "$work/COMPLETE" || exit 1
printf "Backup ready: %s\n" "$work"
EOF
bash -n "$HOME/bin/course-backup.sh"The script makes a unique directory, preventing two runs from choosing the same archive name. Only archives with a final COMPLETE marker are treated as completed backups. Failed runs may leave partial directories for investigation; do not count them as usable recovery points. It records creation time but does not automatically delete older backups.
Test creation, integrity, and restoration separately
lab=$(mktemp -d "$HOME/linux-ops-lab.XXXXXX")
mkdir "$lab/site" "$lab/backups" "$lab/restored"
printf "site version 1\n" > "$lab/site/index.html"
bash "$HOME/bin/course-backup.sh" "$lab/site" "$lab/backups"
find "$lab/backups" -maxdepth 2 -type f -printSet the variable below to the directory printed by this run. Checksum and archive-list success do not prove application recovery: compare restored contents and separately test the actual web service. Replace the placeholder rather than copying it literally.
backup=ACTUAL_BACKUP_DIRECTORY
test -f "$backup/COMPLETE"
(cd "$backup" && sha256sum -c data.tar.gz.sha256)
tar -xzf "$backup/data.tar.gz" -C "$lab/restored"
diff -r "$lab/site" "$lab/restored"Expect a COMPLETE marker, an OK checksum, and no diff differences. Restore into an empty directory and validate required permissions, ownership, and labels. Repeat checks after transferring to independent storage. Apply retention rules that preserve required recovery points rather than deleting old copies indiscriminately when space becomes tight.
Create a focused HTTP and storage health check
The example requires curl, df, and awk. It accepts a URL and a minimum free-space threshold in KiB and treats HTTP 200–299 as healthy. Adjust the criterion if redirects or authentication are expected. It does not validate page contents, database health, or every application function.
cat > "$HOME/bin/course-health.sh" <<'EOF'
#!/usr/bin/env bash
set -u
set -o pipefail
if (( $# != 3 )); then
printf "Usage: %s URL MIN_FREE_KIB STORAGE_PATH\n" "$0" >&2; exit 2
fi
url=$1
minimum=$2
path=$3
[[ "$minimum" =~ ^[0-9]+$ ]] || exit 2
if ! status=$(curl -sS -o /dev/null -w "%{http_code}" --connect-timeout 5 --max-time 15 "$url"); then
printf "HTTP transport failed\n" >&2; exit 1
fi
[[ "$status" =~ ^2[0-9][0-9]$ ]] || { printf "HTTP status=%s\n" "$status" >&2; exit 1; }
available=$(df -Pk -- "$path" | awk 'NR==2 {print $4}') || exit 1
[[ "$available" =~ ^[0-9]+$ ]] || exit 1
(( 10#$available >= 10#$minimum )) || { printf "Low storage: %s KiB\n" "$available" >&2; exit 1; }
printf "OK http=%s free_kib=%s\n" "$status" "$available"
EOF
bash -n "$HOME/bin/course-health.sh"
bash "$HOME/bin/course-health.sh" http://127.0.0.1/ 102400 "$HOME"The 102400 KiB threshold is about 100 MiB and is an illustrative value, not an operating recommendation. Choose thresholds from growth rate and recovery-space needs and inspect inodes separately. For a name-based Nginx site, check the actual domain or appropriate Host routing; a 200 from the default site does not validate the intended site.
Schedule the job and detect failures independently
Reuse the user service/timer structure from lesson 29, with ExecStart invoking /bin/bash %h/bin/course-backup.sh ACTUAL_SOURCE ACTUAL_BACKUP_PATH. Quote arguments according to systemd syntax if paths contain spaces and test the command manually first. Choose a schedule from recovery requirements and keep health checks in a separate unit.
Inspect the next trigger, service exit status, and last completed-backup time separately. A same-server monitor stops when the server stops, so an external monitor should check the real URL and backup freshness. Test any email or messaging alert path and define ownership and recovery actions for each actionable failure.
Exercise failure paths and remove the lab
Change one condition at a time: missing backup source, nonexistent HTTP resource, and an intentionally excessive free-space threshold. Restore each original value and confirm that incomplete archives are not reported as successes. Disable lab timers before removing only the scripts and work directories created for this exercise.
