Rocky Linux 10 Tutorial / Back up and verify restored files and services

Backup in Rocky Linux 10 is not just about storing files together. After restoring to a separate path, you must check the contents, ownership and permissions, and actual service behavior to determine if the backup can be used in the event of a failure. In this tutorial, we carry out this process completely in a small practice environment that does not overwrite operational files.

Deciding what to back up and how far to restore

First, write down "What do we need to have if the entire server disappears to rebuild the service?" Even if you can reinstall the operating system, user-uploaded data, databases, custom configurations, and secret keys cannot be recovered by the installation program. On the other hand, simply copying running processes or temporary caches as files does not necessarily reproduce a normal state.

Target Information to leave together Points to note
Settings directly modified Configuration files, package versions, application order You should be able to reapply only the necessary changes on top of the default settings of the new version.
User·Service data Contents, numeric UID·GID, permissions, ACL, and necessary extended attributes Even if the username is the same, the owner differs if the numeric ID on the new server is different.
Database Dump or consistent backup supported by the DB Simply packaging the files of a running DB with tar is not considered a general proper backup.
Certificates·Secrets·Decryption keys Access rights and key recovery procedure The backup itself is also sensitive data. If you lose the key, you may not be able to read the encrypted backup.
Reconstruction Information Intended settings for storage, service units, firewall, and SELinux This is distinguished from a method that overwrites all files on the current server to a new server at once.

Recovery Point Objective (RPO) is the time range of data loss that can be tolerated during a failure, and Recovery Time Objective (RTO) is the allowable time to restore the service. For example, if requirements state "maximum one day of data loss, service recovery within a few hours," not only backup frequency but also the time to obtain offsite copies and perform restoration tests must be planned. These numbers are examples and the same standards do not apply to all services.

Backups stored on the same disk as the original are useful for practicing file deletion recovery, but they are not separate copies against disk failure. Prepare separate copies on different devices or remote storage with distinct deletion/modification permissions, and designate retention periods and restoration personnel. Simply synchronizing may delete files from the backup that were deleted from the original, so also check whether past point-in-time retention is needed. Basic relative path backup and restoration methods are also explained in Rocky Linux Backup and Restore Documentation .

Creating a practical original that contains settings and user data

Use the dedicated practice VM. Here, source/etc and source/home are directories that mimic the actual /etc and /home, and do not replace system settings. The source lab executed the core backup and restore operations with Rocky Linux 10.2, GNU tar 1.35, ACL tools 2.4.0, attr 2.6.0, Python 3.12.15, and SELinux Enforcing.

lfstudent and labops are accounts created in previous user exercises. In the practice, they had UIDs 1000 and 1001, respectively. In other environments, refer to id results below. ACL (Access Control List) provides information that grants permissions to specific users in addition to the default owner/group permissions, and extended attributes are additional name/value information attached to files.

Switch to the administrator shell and execute the commands below in order. If existing practice paths exist, execution will stop, so do not re-execute without checking. If DNF produces errors when installing missing packages, resolve the cause first.

sudo -i
dnf install -y tar acl attr python3 curl
id lfstudent
id labops
ss -lnt 'sport = :18029'
test ! -e /srv/lf-course29 && test ! -e /var/backups/lf-course29

For port checks, there should be no listener, and the final check should return exit status 0. If even one does not match, stop here. The following block enables set -e so an unexpected command failure stops the sequence. Confirm the prechecks before running it.

set -eu
umask 077
install -d -m 755 /srv/lf-course29
install -d -m 700 /var/backups/lf-course29
install -d -m 755 /srv/lf-course29/source \
  /srv/lf-course29/source/etc /srv/lf-course29/source/etc/lf-app \
  /srv/lf-course29/source/home
install -d -o lfstudent -g lfstudent -m 750 \
  /srv/lf-course29/source/home/lfstudent
printf 'course=Rocky Linux 10\n' > /srv/lf-course29/source/etc/lf-app/app.conf
chmod 644 /srv/lf-course29/source/etc/lf-app/app.conf
printf 'LF29 restored user data\n' > /srv/lf-course29/source/home/lfstudent/document.txt
chown lfstudent:lfstudent /srv/lf-course29/source/home/lfstudent/document.txt
chmod 640 /srv/lf-course29/source/home/lfstudent/document.txt
setfacl -m u:labops:r-x /srv/lf-course29/source/home/lfstudent
setfacl -m u:labops:r-- /srv/lf-course29/source/home/lfstudent/document.txt
setfattr -n user.lf_course -v backup29 \
  /srv/lf-course29/source/home/lfstudent/document.txt
ln /srv/lf-course29/source/home/lfstudent/document.txt \
  /srv/lf-course29/source/home/lfstudent/document-hard.txt
ln -s document.txt /srv/lf-course29/source/home/lfstudent/document-link.txt
restorecon -RF /srv/lf-course29/source

document-hard.txt is a hard link that shares the same file content, and document-link.txt is a symbolic link pointing to the target path. Later, we check not only that the contents are the same but also this relationship. The last command applies the default SELinux label for the path policy only to the newly created practice originals. It does not forcibly re-label the entire actual operating directory.

Back up including attributes and verify the checksum

The practice originals are backed up in a state where they are no longer modified. In an actual service, procedures to stop writing, backup functions supported by the application, or snapshots that ensure consistency may be required. Even if checksums match, backups containing data from different points in time for each file may not be in a consistent state for business purposes.

tar --acls --xattrs --selinux --numeric-owner \
  -cpf /var/backups/lf-course29/backup.tar \
  -C /srv/lf-course29/source .
cd /var/backups/lf-course29
sha256sum backup.tar > backup.tar.sha256
sha256sum -c backup.tar.sha256
tar --acls --xattrs --selinux --numeric-owner -tvf backup.tar

-c creates an archive, the value following -f is the archive file name, and -C changes the reference location to that directory. Therefore, the storage path becomes a relative path as in ./etc/..., making it easier to safely verify from another location. The backup files were placed outside the original directory. -p is mainly an option to preserve permissions during restoration, so this alone does not replace the preservation of ACLs or extended attributes.

--acls, --xattrs, --selinux each specify support for the corresponding metadata. The restoration of extended attributes in GNU tar targets the user namespace by default, while SELinux labels are handled with a separate option. In this example, user.lf_course and SELinux labels are checked separately. The scope is not extended to forcibly restore other security attributes. The GNU tar extended-attribute documentation explains these namespaces. Check tar --help for support in your installed build.

If everything is normal, backup.tar: OK will be displayed, and the list will show the configuration files, user files, and the two links. --numeric-owner handles ownership based on numerical IDs instead of names. If the same number is assigned to a different user on a new server, you should first match the accounts before restoring as is.

SHA-256 checksums verify whether the current file matches the bytes written. Placing the checksum file in a location where it could also be tampered with cannot prove the source's authenticity. It should be combined with checksums delivered through trusted channels or with separate signatures and access controls.

Restore first to an empty separate path

Check that only the expected relative paths and link targets are present in the list. Do not extract archives from unknown sources with administrative privileges. Stop if there are absolute paths, paths escaping to parent directories, suspicious links, or device files. Simply checking the list does not make a malicious archive safe. In this tutorial, only small practice backups that you created yourself are used as restoration targets.

test ! -e /srv/lf-course29/restore
install -d -m 755 /srv/lf-course29/restore
RESTORE_ENTRIES=$(find /srv/lf-course29/restore -mindepth 1 -print -quit)
test -z "$RESTORE_ENTRIES"
cd /var/backups/lf-course29
sha256sum -c backup.tar.sha256
tar --acls --xattrs --selinux --numeric-owner \
  --same-owner --same-permissions -xpf backup.tar \
  -C /srv/lf-course29/restore

-x selects extraction; --same-owner and --same-permissions request ownership and permission restoration. Here, only restore trusted backups you created yourself into an empty practice directory. Do not rename it to -C /, as that could affect actual system files. If there is already a restored copy, do not automatically empty or overwrite it; prepare a new directory.

Preserving SELinux labels and applying labels according to the policy of the new path are separate tasks. When checking restoration accuracy in the same environment, compare the stored labels; when deploying to a different service path, review and apply the file context rules of that server. Changing directly to restorecon makes it difficult to first compare whether the original labels were preserved.

Compare file contents along with permissions, ACLs, and links together

The following inspection compares the entire subtree of this exercise. It checks the SHA-256 of each file, numeric ownership, permissions, ACL, readable extended attributes, and the target of symbolic links, and also verifies the hard link relationship of two regular files. SHA-256 calculations use the Python standard library hashlib . This is not a tool to replace backup products for large-scale operational data, but validation code for this exercise where file names, number, and contents are predetermined.

python3 - <<'PY'
import os, stat, hashlib, subprocess
from pathlib import Path
base = Path('/srv/lf-course29')
def manifest(root):
    result = {}
    for p in sorted([root] + list(root.rglob('*'))):
        s = p.lstat()
        row = {'mode': stat.S_IMODE(s.st_mode), 'uid': s.st_uid,
               'gid': s.st_gid, 'type': stat.S_IFMT(s.st_mode)}
        row['xattrs'] = {
            k: os.getxattr(p, k, follow_symlinks=False)
            for k in os.listxattr(p, follow_symlinks=False)}
        if p.is_symlink():
            row['target'] = os.readlink(p)
        elif p.is_file():
            row['sha256'] = hashlib.sha256(p.read_bytes()).hexdigest()
        if not p.is_symlink():
            row['acl'] = subprocess.check_output(
                ['getfacl', '-cn', str(p)], text=True)
        result[str(p.relative_to(root))] = row
    return result
before = manifest(base/'source')
after = manifest(base/'restore')
if before != after:
    raise SystemExit('FAIL: restored content or metadata differs')
for name in ('source', 'restore'):
    d = base/name/'home/lfstudent'
    if (d/'document.txt').stat().st_ino != (d/'document-hard.txt').stat().st_ino:
        raise SystemExit('FAIL: hard link relation differs')
print('PASS: restored content and metadata match; paths:', len(before))
PY
getfacl -cn /srv/lf-course29/restore/home/lfstudent/document.txt
getfattr -n user.lf_course /srv/lf-course29/restore/home/lfstudent/document.txt
ls -lZ /srv/lf-course29/restore/home/lfstudent
runuser -u labops -- cat /srv/lf-course29/restore/home/lfstudent/document.txt

In the recorded source setup, PASS and the number of paths 9 are displayed.

ACL showed user:1001:r-- of the exercise VM, and extended attributes showed user.lf_course="backup29". The link count of the two regular files was 2, and the SELinux type was the same in this exercise path as var_t. The result read as labops was LF29 restored user data. Instructions like "Removing leading '/'" in getfacl mean that leading slashes were removed in the displayed path, and in itself, this does not indicate a restoration failure.

If you include the file modification time in the requirements for operational recovery, compare it separately. This check does not verify the identicalness of access time, creation time, or inode number itself, nor does it verify all types of special files. It is normal for inode numbers to differ from the original in a new filesystem, and here we only check whether two hard links within the restored copy share the same inode.

Run the service with the restored data and check for errors.

Administrators can read files that normal users cannot due to permissions. Therefore, just because cat succeeds in the administrator shell does not mean the service recovery is complete. Below is a full Python program for practice that reads the configuration and user data and returns it as HTTP responses. It is not an externally public web server and only listens on the loopback address.

cat > /srv/lf-course29/app.py <<'PY'
import json, sys
from pathlib import Path
from http.server import BaseHTTPRequestHandler, HTTPServer
root = Path(sys.argv[1])
class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        try:
            result = {'config': (root/'etc/lf-app/app.conf').read_text().strip(),
                      'data': (root/'home/lfstudent/document.txt').read_text().strip()}
            body = (json.dumps(result) + '\n').encode()
            self.send_response(200)
            content_type = 'application/json'
        except (OSError, UnicodeError):
            body = b'Cannot read application data\n'
            self.send_response(500)
            content_type = 'text/plain; charset=utf-8'
        self.send_header('Content-Type', content_type)
        self.end_headers()
        self.wfile.write(body)
HTTPServer(('127.0.0.1', 18029), Handler).serve_forever()
PY
chmod 644 /srv/lf-course29/app.py
systemd-run --unit=lf-course29-restore --uid=lfstudent \
  --property=Type=exec /usr/bin/python3 /srv/lf-course29/app.py \
  /srv/lf-course29/restore
sleep 1
curl --fail --silent http://127.0.0.1:18029/

systemd-run creates a temporary service here for a one-time test and limits the execution account with --uid=lfstudent. A normal response is as follows.

{"config": "course=Rocky Linux 10", "data": "LF29 restored user data"}

If the connection is refused, check for program errors and port conflicts with systemctl status lf-course29-restore.service and journalctl -u lf-course29-restore.service -n 30 --no-pager first. No firewall opening or port forwarding is needed for this loopback practice. For actual services, test according to the environment, including the service account, SELinux domain, secret information delivery, and external dependencies.

To see why permission verification is necessary, temporarily make one user file in the restored copy unreadable. Do not touch the original and normal backup.

p=/srv/lf-course29/restore/home/lfstudent/document.txt
getfacl -cn "$p" > /var/backups/lf-course29/document-acl-before.txt
chmod 000 "$p"
curl --silent --write-out '\nHTTP %{http_code}\n' http://127.0.0.1:18029/
chmod 640 "$p"
getfacl -cn "$p" > /var/backups/lf-course29/document-acl-after.txt
cmp /var/backups/lf-course29/document-acl-before.txt \
  /var/backups/lf-course29/document-acl-after.txt
curl --fail --silent http://127.0.0.1:18029/

When reading was blocked, Cannot read application data and HTTP 500 appeared, and after restoring the permissions, it recovered to normal JSON and HTTP 200. The comparison before and after ACL also matched. Since hard links share the same file, this permission change also applies to document-hard.txt in the restored copy. After checking the error, be sure to execute the recovery command.

Reject a damaged archive before extraction

To practice a checksum failure, change a single data byte in a copy of the normal backup. The following code records the checksum of the normal file as the expected value for the damaged copy, so the check should fail. The actual original backup and existing checksum are preserved.

python3 - <<'PY'
from pathlib import Path
import hashlib
d = Path('/var/backups/lf-course29')
data = (d/'backup.tar').read_bytes()
offset = data.find(b'LF29 restored user data\n')
if offset < 0:
    raise SystemExit('Expected fixture payload not found')
bad = bytearray(data)
bad[offset] ^= 1
(d/'backup-corrupt.tar').write_bytes(bad)
(d/'corrupt.expected.sha256').write_text(
    hashlib.sha256(data).hexdigest() + '  backup-corrupt.tar\n')
PY
cd /var/backups/lf-course29
if sha256sum -c corrupt.expected.sha256; then
  echo 'Unexpected result: stop and inspect the test'
else
  echo 'Expected checksum failure: do not extract this copy'
fi
sha256sum -c backup.tar.sha256

The damaged copy should be FAILED, and the normal backup should be OK. Checking whether the tar list opens is necessary, but it is not equivalent to a cryptographic check that detects all file content alterations. The source lab did not extract the damaged copy. If a production backup fails its checksum, do not ignore the warning or restore over live data. Investigate the transfer and storage path, then select another verified copy.

Apply the recovery checks to production and finish the lab

In actual fault recovery, first preserve the current failure state and place the verified data during the maintenance time when the service is stopped. After matching ownership and applying the SELinux context suitable for the path, check the settings, start the service, and verify key functions such as login, query, and write, as well as error logs. If you do not define the point when user data will be overwritten and when recovery resumes, you may lose new incoming data or process it redundantly during recovery.

If the recovery fails, stop the service, revert to the previously preserved settings and data, and investigate the cause. If a service references both the database and files, the timing of both data must also match. The small text data and loopback HTTP success in this exercise should not be interpreted as results for actual database recovery or validation of an entire external service.

After finishing the practice service, check that the port is closed. Keep the data and normal backups for re-verification and do not automatically delete directories.

systemctl stop lf-course29-restore.service
ss -lnt 'sport = :18029'
cd /var/backups/lf-course29
sha256sum -c backup.tar.sha256
getenforce
exit

If there is no listener on the port check and the normal backup is OK, and SELinux is Enforcing, then this is the end state of this exercise. Production backups should periodically test the same restoration procedures on copies stored in different locations, and record the successful restoration time, backups used, time taken, and results of function checks. Backup success notifications and actual recoverability should be verified separately.