Rocky Linux 10 Tutorial / Build a restricted service and test backup recovery

Comprehensive practice in Rocky Linux 10 server operation does not end with just starting a service. You need to run it with an account that has only the necessary permissions, back up normal data, detect and restore intentional corruption, and check if the data persists after a restart. In this tutorial, we create a local counter service and connect the entire process in one go.

Practice Objectives and Safe Preparation Scope

The service reads the current value with a GET /counter request and increments it by 1 with a POST /counter request. The value is not kept only in memory but also stored in a separate state file. The goal is to back up the normal value 2, damage the data intentionally, restore 2 from the backup, increment it to 3 again, and check whether it persists after the service restarts.

This is carried out on a dedicated Rocky Linux 10 VM. The source lab environment was Rocky 10.2, systemd 257, Python 3.12.15, GNU tar 1.35, with SELinux Enforcing. The Python standard HTTP server is for this local practice, and the Python official HTTP server documentation does not recommend its use in production. This example, lacking authentication and TLS, is not exposed to external services.

Below is the process in the admin shell to check for conflicts with new accounts, paths, and ports. If existing items are found, do not delete or reuse them, and stop. The set -e in the shell ensures that it does not continue after unexpected failures, and a separate if is used for checks where failure is expected.

sudo -i
set -euo pipefail
umask 077
rpm -q python3 tar policycoreutils
test ! -e /etc/systemd/system/lf-course32.service
test ! -e /etc/lf-course32
test ! -e /srv/lf-course32
test ! -e /var/lib/lf-course32
test ! -e /var/backups/lf-course32
if getent passwd lfcourse32; then exit 1; fi
if getent group lfcourse32; then exit 1; fi
LISTENERS=$(ss -H -lnt 'sport = :18040')
test -z "$LISTENERS"
systemctl is-active sshd firewalld
getenforce

Required packages and curl commands must be prepared. Existing admin access and firewalls are maintained, and no additional port forwarding or external firewall openings are made. All /etc/lf-course32, /srv/lf-course32, /var/lib/lf-course32, and backup paths used going forward are dedicated to this practice.

Separate the service account, configuration and state

Create a non-login system account lfcourse32 so that the service does not run under the administrator or a personal login account. Even if the login shell is nologin, systemd can run programs under this account. The administrator owns the settings and programs, while the service only reads them and records work data only in the state directory.

useradd --system --user-group --home-dir /var/lib/lf-course32 \
  --no-create-home --shell /sbin/nologin lfcourse32
install -d -o root -g lfcourse32 -m 750 /etc/lf-course32
install -d -o root -g root -m 755 /srv/lf-course32
install -d -m 700 /var/backups/lf-course32
printf '{"bind":"127.0.0.1","port":18040,"state":"/var/lib/lf-course32/counter.json"}\n' \
  > /etc/lf-course32/app.json
chown root:lfcourse32 /etc/lf-course32/app.json
chmod 640 /etc/lf-course32/app.json

The following are all the necessary programs. When saving, write to a temporary file in the same directory first and then replace the name. This is a way to reduce the possibility of exposing intermediate content in this single-process exercise, and it is not a storage device that guarantees DB transactions or persistence under all power outage situations.

cat > /srv/lf-course32/app.py <<'PY'
import json, os, sys, tempfile
from pathlib import Path
from http.server import BaseHTTPRequestHandler, HTTPServer
config = json.loads(Path(sys.argv[1]).read_text())
state = Path(config['state'])
def save(value):
    fd, name = tempfile.mkstemp(dir=state.parent, prefix='.counter-')
    try:
        with os.fdopen(fd, 'w') as stream:
            json.dump({'count': value}, stream)
            stream.write('\n')
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(name, state)
    finally:
        if os.path.exists(name):
            os.unlink(name)
if not state.exists():
    save(0)
class Handler(BaseHTTPRequestHandler):
    def respond(self, increment=False):
        if self.path != '/counter':
            self.send_error(404)
            return
        try:
            count = json.loads(state.read_text())['count']
            if type(count) is not int or count < 0:
                raise ValueError('Invalid counter')
            if increment:
                count += 1
                save(count)
            body = (json.dumps({'count': count}) + '\n').encode()
            self.send_response(200)
        except (OSError, ValueError, KeyError):
            body = b'{"error":"Counter data unavailable"}\n'
            self.send_response(503)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(body)
    def do_GET(self):
        self.respond()
    def do_POST(self):
        self.respond(True)
HTTPServer((config['bind'], config['port']), Handler).serve_forever()
PY
chmod 644 /srv/lf-course32/app.py

If the state file does not exist from the beginning, create 0. If the contents of an existing file are incorrect, return 503 in response to the request and do not initialize arbitrarily. This configuration is to notify the administrator that recovery is needed instead of silently replacing corrupted data with new data during a failure.

However, in this example, the absence of a file is treated as initial creation, so if the file is deleted and restarted, 0 can be created. In actual operation, the first initialization and normal startup should be distinguished, and missing existing state files should be treated as an error requiring recovery instead of automatic initialization. The failures in this exercise are limited to incorrect content, not file deletion.

Run as a restricted systemd service

The following unit file sets the execution account, file access scope, and state directory management together. StateDirectory=lf-course32 creates the /var/lib/lf-course32 for the system service and prepares ownership according to the execution account. Therefore, the program does not need to write data to the home directory.

cat > /etc/systemd/system/lf-course32.service <<'UNIT'
[Unit]
Description=Course32 counter recovery lab
After=network.target
[Service]
Type=exec
User=lfcourse32
Group=lfcourse32
ExecStart=/usr/bin/python3 /srv/lf-course32/app.py /etc/lf-course32/app.json
StateDirectory=lf-course32
StateDirectoryMode=0750
UMask=0077
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
CapabilityBoundingSet=
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
Restart=on-failure
RestartSec=2
[Install]
WantedBy=multi-user.target
UNIT
restorecon -RF /etc/lf-course32 /srv/lf-course32 \
  /etc/systemd/system/lf-course32.service
systemd-analyze verify /etc/systemd/system/lf-course32.service
systemctl daemon-reload
systemctl start lf-course32.service
Configuration Roles and Scope
User·Group Runs under a dedicated account. In the practice VM, UID·GID 994 is assigned, and it may differ on other servers.
NoNewPrivileges Restricts the paths through which the executed program can gain additional privileges.
CapabilityBoundingSet= Leaves the service permission scope of Linux capabilities, which divide administrator privileges by function, empty.
ProtectSystem=strict Makes most of the filesystem read-only. Exceptions for writable directories such as state directories should be reviewed.
StateDirectory·PrivateTmp StateDirectory provides persistent service data; PrivateTmp provides private temporary directories. Review these writable exceptions alongside ProtectSystem=strict.
ProtectHome Prevents the service from accessing user homes, admin homes, etc.
RestrictAddressFamilies Limits the socket address families that can be used. This is not a firewall setting that restricts destination IPs or internet access itself.

Applying restrictions that do not fit the function all at once may cause normal services to fail. Check each option against the Rocky Linux systemd hardening guide and the systemd 257 execution settings for the installed version. This setting alone does not eliminate all attacks or operational risks.

Check the process state and actual requests separately

The successful start of Type=exec means that the program has started successfully, but it does not mean that it is ready to handle HTTP requests. During the first practice, the connection was sometimes refused because the request was too fast immediately after starting. As shown below, we wait for readiness limited to read requests and verify normal values.

systemctl show lf-course32.service -p User -p Group \
  -p NoNewPrivileges -p ProtectSystem -p StateDirectory -p MainPID
pid=$(systemctl show lf-course32.service -p MainPID --value)
grep -E '^(Uid|Gid|CapEff|NoNewPrivs):' "/proc/$pid/status"
curl --retry 5 --retry-connrefused --retry-delay 1 --max-time 3 \
  --fail --silent http://127.0.0.1:18040/counter
curl --fail --silent -X POST http://127.0.0.1:18040/counter
curl --fail --silent -X POST http://127.0.0.1:18040/counter
stat -c '%U %G %a %n' /var/lib/lf-course32/counter.json

The first read value should be 0, and the value after incrementing twice should be 1 and 2, respectively. In the source lab, the process UID·GID was 994, the effective capability value was 0, and NoNewPrivs was 1. The owner of the state file was lfcourse32, and the permissions were 600. The result shows that we checked the response together with the actual file to confirm that the new value was stored.

Retry attempts for waiting for readiness were applied only to read requests. POST requests that change values can be applied twice if they are repeated unconditionally without receiving a response. In such cases, check the current value first to verify the result. If the connection still fails, check systemctl status lf-course32.service, journalctl -u lf-course32.service, and the configuration and file permissions.

Back up the working service in a consistent state

After stopping the write service, back up the program, configuration, state, and unit files together. In this program, only one process writes, so stopping the service is the write stop point. For services used by multiple servers or databases together, a separate method for consistent backup of that product must be defined.

systemctl stop lf-course32.service
tar --acls --xattrs --selinux --numeric-owner \
  -cpf /var/backups/lf-course32/backup.tar -C / \
  etc/lf-course32 srv/lf-course32 var/lib/lf-course32 \
  etc/systemd/system/lf-course32.service
cd /var/backups/lf-course32
sha256sum backup.tar > backup.tar.sha256
sha256sum -c backup.tar.sha256
tar -tvf backup.tar
sha256sum /etc/lf-course32/app.json /srv/lf-course32/app.py \
  /etc/systemd/system/lf-course32.service > application.sha256
sha256sum /var/lib/lf-course32/counter.json > counter-before.sha256

The checksum is OK, and only the expected practice paths should be in the list. The backup contains the correct value 2. application.sha256 is used to check whether settings and programs were changed during data corruption or recovery, and counter-before.sha256 is used to confirm that the state file has returned exactly the same as at the backup point.

This backup is placed on the same VM to practice the recovery flow. Actual operational backups are preserved in separate locations, considering total disk/server failure and original deletion permissions. Simply having a checksum does not prove the source of the backup or the consistency of operational data. Methods to check file attributes and restore to separate paths are covered in detail in Settings/User Data Backup and Actual Restoration Verification course .

Create data corruption and distinguish it from the normal state.

Restart the service and first read value 2 normally. Then, change only this practice-specific state file to incorrect content. Do not change actual operational data or other service files.

systemctl start lf-course32.service
curl --retry 5 --retry-connrefused --retry-delay 1 --max-time 3 \
  --fail --silent http://127.0.0.1:18040/counter
printf 'broken counter data\n' > /var/lib/lf-course32/counter.json
curl --silent --output failure-response.txt --write-out 'HTTP %{http_code}\n' \
  http://127.0.0.1:18040/counter
cat failure-response.txt
systemctl is-active lf-course32.service
journalctl -u lf-course32.service -n 20 --no-pager

The expected response is HTTP 503 and the following JSON.

{"error":"Counter data unavailable"}

At this time, systemctl is-active was still active. The program did not terminate because the data to handle the request was incorrect. Restart=on-failure is a restart setting for termination failure and does not automatically restore corrupted work data to a normal backup. This is why checks on process, port, HTTP response, and data are all necessary.

The logs allow comparison of 200 and 503 responses for the same path request. In actual failures, the modification time, last normal response, and ownership, permissions, and format of the data are examined together. Data is not initialized before confirming whether the unreadable cause is a permissions issue or incorrect content.

Preserve corrupted data and restore it to normal values

Stop the service to prevent writes during restoration, and keep corrupted data under a separate name. Extract only the dedicated counter state file from the trusted backup created in this lab. Do not add other paths or extract an unverified archive with administrator privileges.

systemctl stop lf-course32.service
test ! -e counter-corrupt.saved
cp -a /var/lib/lf-course32/counter.json counter-corrupt.saved
sha256sum -c backup.tar.sha256
tar --acls --xattrs --selinux --numeric-owner \
  --same-owner --same-permissions -xpf backup.tar -C / \
  var/lib/lf-course32/counter.json
sha256sum -c counter-before.sha256
stat -c '%U %G %a %n' /var/lib/lf-course32/counter.json
systemctl start lf-course32.service
curl --retry 5 --retry-connrefused --retry-delay 1 --max-time 3 \
  --fail --silent http://127.0.0.1:18040/counter

The checksum of the state file should be OK, the owner should be lfcourse32, permissions should be 600, and the response value should return 2. Distinguish and verify between the result of simply restarting the service and the result of actually restoring a normal backup.

After restoration, increment the value once to verify that writes work, then restart the service and read it again.

curl --fail --silent -X POST http://127.0.0.1:18040/counter
systemctl restart lf-course32.service
curl --retry 5 --retry-connrefused --retry-delay 1 --max-time 3 \
  --fail --silent http://127.0.0.1:18040/counter
sha256sum -c application.sha256

Immediately after the increase and after the restart, {"count": 3} appeared, and the checksums of the program, settings, and unit files remained unchanged. This result confirms the file persistence after the service restart and does not reflect the results of testing automatic VM startup for new accounts/services or power outage recovery in this comprehensive exercise.

Record the verified scope and handover information

This practice started the service manually without enabling boot automatic start. After verification is completed, stop it and check the inactive state and closed ports. Retain the dedicated non-login account, program, configuration, state and backup for review and further restoration drills. Other accounts or existing services are not subject to cleanup.

systemctl stop lf-course32.service
systemctl is-enabled lf-course32.service || true
ss -lnt 'sport = :18040'
cd /var/backups/lf-course32
sha256sum -c /var/backups/lf-course32/backup.tar.sha256
systemctl is-active sshd firewalld
getenforce
exit

In the source VM, the unit ended with disabled and no listener on port 18040. Administrative access and the firewall were maintained, and SELinux was Enforcing. The source checks found the original fstab, network profile and firewall unchanged. Existing account and group entries were preserved while adding the dedicated service account and group.

For operational handover, leave the actually verified contents and scope as follows.

  • Executable file, configuration, status path, execution account, service name, receiving address, and port
  • How to check normal requests, actual data write results, and readiness for startup
  • Location, checksum, point-in-time of used backup, and the sequence to stop services
  • Preserving damaged data, restoring, verifying functionality after resumption, and response methods in case of failure
  • Conditions like unverified external connections, rebooting, multiple writes, power outages, and additional personnel for confirmation

Operating procedures can be used only if functioning services, restricted execution privileges, recoverable backups, and actual functionality verification are connected. Recording the success status of each step separately allows you to accurately assess scenarios like 'running but request fails' or 'file restored but write does not work'.