Debian 13 Tutorial / Capstone Lab: Deploy a Service, Detect Corrupt Data and Restore a Backup
To verify the operation of a Debian server, you need to check not only whether the service is running but also whether it correctly reads and writes data, and whether new writes are maintained after restoring corrupted data from backups. In this comprehensive practice, you will deploy the counter service with a dedicated account and sequentially perform normal backups, limited data corruption, HTTP error detection, and actual restoration.
Define the Lab Environment and Success Criteria
Use a separate virtual machine (VM) with Debian 13, Python 3.13, and systemd 257. If Python 3, curl, or tar are not available, prepare them with sudo apt install python3 curl tar. Execute with an administrative account that has sudo privileges, and do not apply to servers already using any of the specified paths, the account or port8086.
The HTTP service only accepts requests at 127.0.0.1:8086. GET / reads the current value, and POST /increment with no body increments the value by one. This is for local learning, not an operational product implementing authentication, external Transport Layer Security (TLS), or concurrent writes by multiple processes. Also check the scope of Python HTTP server documentation.
| Steps | Success Criteria |
|---|---|
| Deployment and Writing | With a dedicated account: 0 → 1 → 2, and still 2 after service restart |
| Normal Backup | Generate archive and hash in stopped state, restore separate empty directories, compare contents and permissions |
| Damage Detection | Processes and ports are maintained but both GET and POST return 503, corrupted files are not automatically initialized |
| Operation after Restoration | Restore 2 from actual backup → write new 3 → after restart 3 → write new 4 |
The commands below are the sequence for executing a new practice once. Check to avoid overwriting the same backup, and if it fails in the middle, check the state and logs, then resolve the cause of the failed step first. Do not run the initial block again to forcibly initialize normal values.
Deploy the Complete Program Under a Dedicated Account
Before setup, also confirm getent group lf-capstone36 finds no existing group. Stop on any failed prerequisite or creation step rather than reusing an existing account, directory or service. Ensure loopback curl requests bypass any configured proxy.
Program and service units are owned by root, and only the data directory is managed by a dedicated account. Within the read-only system restrictions of the unit, the data path is allowed for ReadWritePaths. In storage methods that create temporary files and replace names, not only write permission for a single file but also write and traversal permissions for the parent directory are required.
sudo bash <<'ROOT'
set -euo pipefail
for p in /opt/lf-capstone36 /var/lib/lf-capstone36 /var/backups/lf-capstone36 /etc/systemd/system/lf-capstone36.service; do test ! -e "$p"; done
if getent passwd lf-capstone36 >/dev/null; then exit 1; fi
test -z "$(ss -H -ltn 'sport = :8086')"
useradd --system --user-group --home-dir /var/lib/lf-capstone36 --no-create-home --shell /usr/sbin/nologin lf-capstone36
install -d -m 755 /opt/lf-capstone36
install -d -m 750 -o lf-capstone36 -g lf-capstone36 /var/lib/lf-capstone36
install -d -m 700 /var/backups/lf-capstone36
cat > /opt/lf-capstone36/app.py <<'PY'
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path
import json
import os
import tempfile
DATA = Path('/var/lib/lf-capstone36/state.json')
def load_count():
value = json.loads(DATA.read_text(encoding='utf-8'))
if (type(value) is not dict or set(value) != {'count'}
or type(value['count']) is not int or value['count'] < 0):
raise ValueError('invalid state')
return value['count']
def save_count(count):
temporary = None
try:
with tempfile.NamedTemporaryFile(
mode='w', encoding='utf-8', dir=DATA.parent,
prefix='state.', delete=False) as stream:
temporary = Path(stream.name)
json.dump({'count': count}, stream, separators=(',', ':'))
stream.write('\n')
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, DATA)
descriptor = os.open(DATA.parent, os.O_RDONLY | os.O_DIRECTORY)
try:
os.fsync(descriptor)
finally:
os.close(descriptor)
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
class Handler(BaseHTTPRequestHandler):
def reply(self, status, value):
body = (json.dumps(value, separators=(',', ':')) + '\n').encode()
self.send_response(status)
self.send_header('Content-Type', 'application/json; charset=utf-8')
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if self.path != '/':
self.reply(404, {'error': 'not found'})
return
try:
count = load_count()
except (OSError, ValueError):
self.reply(503, {'error': 'state unavailable'})
return
self.reply(200, {'count': count})
def do_POST(self):
if self.path != '/increment':
self.reply(404, {'error': 'not found'})
return
if self.headers.get('Content-Length', '0') != '0':
self.reply(400, {'error': 'empty body required'})
return
try:
count = load_count() + 1
save_count(count)
except (OSError, ValueError):
self.reply(503, {'error': 'state unavailable'})
return
self.reply(200, {'count': count})
HTTPServer(('127.0.0.1', 8086), Handler).serve_forever()
PY
cat > /etc/systemd/system/lf-capstone36.service <<'UNIT'
[Unit]
Description=LF capstone counter 36
After=network.target
[Service]
Type=exec
User=lf-capstone36
Group=lf-capstone36
WorkingDirectory=/opt/lf-capstone36
ExecStart=/usr/bin/python3 -B /opt/lf-capstone36/app.py
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/lf-capstone36
MemoryMax=128M
TasksMax=32
Restart=on-failure
RestartSec=2s
[Install]
WantedBy=multi-user.target
UNIT
chmod 644 /opt/lf-capstone36/app.py /etc/systemd/system/lf-capstone36.service
printf '{"count":0}\n' > /var/lib/lf-capstone36/state.json
chown lf-capstone36:lf-capstone36 /var/lib/lf-capstone36/state.json
chmod 600 /var/lib/lf-capstone36/state.json
systemd-analyze verify /etc/systemd/system/lf-capstone36.service
systemctl daemon-reload
systemctl enable --now lf-capstone36.service
sleep .4
test "$(curl -fsS http://127.0.0.1:8086/)" = '{"count":0}'
test "$(curl -fsS -X POST http://127.0.0.1:8086/increment)" = '{"count":1}'
test "$(curl -fsS -X POST http://127.0.0.1:8086/increment)" = '{"count":2}'
systemctl restart lf-capstone36.service
sleep .3
test "$(curl -fsS http://127.0.0.1:8086/)" = '{"count":2}'
systemctl is-enabled lf-capstone36.service
pid=$(systemctl show -p MainPID --value lf-capstone36.service)
ps -o user=,group=,args= -p "$pid"
if runuser -u lf-capstone36 -- test -w /opt/lf-capstone36/app.py; then echo CODE_WRITABLE >&2; exit 1; fi
if runuser -u lf-capstone36 -- test -r /var/backups/lf-capstone36; then echo BACKUP_READABLE >&2; exit 1; fi
systemctl show lf-capstone36.service -p NoNewPrivileges -p ProtectSystem -p ReadWritePaths -p MemoryMax -p TasksMax
echo COUNTER0_TO1_TO2_AND_RESTART_PERSISTENCE_LEAST_PRIVILEGE_PASSED
ROOT
The short startup sleeps are delays, not readiness guarantees. If a request fails, inspect sudo systemctl status lf-capstone36.service and sudo journalctl -u lf-capstone36.service -n 30, then use GET to check the current value. Do not repeat the whole setup or blindly retry POST: a write may have committed even if its response was lost.
If it executes to the end of the block without errors, 0 → 1 → 2 and after restarting the service, maintaining 2 is confirmed. The dedicated account cannot modify code files or read root-only backup directories, and the execution restriction output shows ProtectSystem=strict, NoNewPrivileges=yes, memory 128 MiB, and 32 tasks. These limits are for a small example and should be adjusted to match the measured values of the actual service.
The program uses a JSON (JavaScript Object Notation) parser to check the structure, integer type, and range of the read values. If it encounters an invalid file, it returns 503 instead of resetting to 0. Automatically initializing to 0 could lose evidence to recover and existing values, and hide errors as normal responses.
When saving, create a temporary file in the same data directory using NamedTemporaryFile, and after flush and fsync, replace the name with os.replace. The directory is also synchronized, but this exercise did not test power outages or virtual disk cache loss. Do not consider atomic name replacement and durability against power failure on all storage devices as providing the same guarantee.
This HTTPServer handles requests sequentially within one server process. Synchronization for cases where multiple processes modify the same file or external tools write simultaneously is not implemented. Also, increment requests change the value each time they are repeated, so blindly resending because a response was missed can increase it twice. In actual operations, you should consider a storage method with transaction and duplicate request handling.
The native lab verified the enabled state and service restart using systemctl. Since automatic start after VM reboot was not separately tested in this comprehensive exercise, before actual handover, also perform a boot test while ensuring console and recovery means. The meaning of account, path, and protection settings can be checked in systemd execution environment manual.
Back Up the Stopped Service and Test an Empty-directory Restore
Stop the service and back up the counter while its value is 2. If sending additional requests changes the value, check why it changed and set the intended reference value. Do not overwrite data with 2 if the following check fails.
sudo bash <<'ROOT'
set -euo pipefail
dest=/var/backups/lf-capstone36
test ! -e "$dest/service.tar"
systemctl stop lf-capstone36.service
test "$(cat /var/lib/lf-capstone36/state.json)" = '{"count":2}'
sha256sum /var/lib/lf-capstone36/state.json > "$dest/state-before.sha256"
stat -c '%u:%g:%a' /var/lib/lf-capstone36/state.json > "$dest/state-mode.txt"
tar --acls --xattrs --numeric-owner -cpf "$dest/service.tar" -C / opt/lf-capstone36 var/lib/lf-capstone36 etc/systemd/system/lf-capstone36.service
cd "$dest"
sha256sum service.tar > service.sha256
sha256sum -c service.sha256
mkdir -m 700 restore-test
tar --acls --xattrs --numeric-owner --same-owner -xpf service.tar -C restore-test
cmp /opt/lf-capstone36/app.py restore-test/opt/lf-capstone36/app.py
cmp /var/lib/lf-capstone36/state.json restore-test/var/lib/lf-capstone36/state.json
test "$(stat -c '%u:%g:%a' restore-test/var/lib/lf-capstone36/state.json)" = "$(cat state-mode.txt)"
systemctl start lf-capstone36.service
sleep .3
test "$(curl -fsS http://127.0.0.1:8086/)" = '{"count":2}'
echo STOPPED_BACKUP_CHECKSUM_AND_EMPTY_DIRECTORY_RESTORE_PASSED
ROOT
service.tar: OK is an archive integrity check. Then, compare the bytes of the extracted programs and data in the empty restore-test as well as the data file UID, GID, and permissions. This is a stronger verification than just the existence of the backup file but is not yet a restoration from failure in the actual service path. That process is carried out separately below.
The archive includes only the programs, data, and units of this exercise. Because /var/backups is on the same disk, it is not an independent backup against total disk loss. For actual operational data, also prepare a stored copy with separated storage and access rights and check it at the designated restoration interval. You can refer to the options of tar in GNU tar Debian Manual.
Corrupt Only the Lab Data and Detect HTTP Errors
The following block deliberately changes only the /var/lib/lf-capstone36/state.json you just created to a semantically invalid state: valid JSON with a string where an integer is required. Execute this only after passing the normal backup check and do not replace the path with operational files. Do not experiment with killing processes or filling up the disk.
sudo bash <<'ROOT'
set -euo pipefail
dest=/var/backups/lf-capstone36
printf '{"count":"broken"}\n' > /var/lib/lf-capstone36/state.json
sha256sum /var/lib/lf-capstone36/state.json > "$dest/broken-state.sha256"
test "$(curl -sS -o "$dest/broken-get.txt" -w '%{http_code}' http://127.0.0.1:8086/)" = 503
test "$(curl -sS -X POST -o "$dest/broken-post.txt" -w '%{http_code}' http://127.0.0.1:8086/increment)" = 503
sha256sum -c "$dest/broken-state.sha256"
test "$(cat "$dest/broken-get.txt")" = '{"error":"state unavailable"}'
systemctl is-active lf-capstone36.service
ss -ltn 'sport = :8086'
journalctl -u lf-capstone36.service -n 8 --no-pager
echo ACTIVE_PROCESS_OPEN_PORT_BUT_GET_AND_POST_503_BROKEN_DATA_UNCHANGED
ROOT
Since count was a string in the JSON, both GET and POST should return 503. curl -f fails in the error response, so in this section, we omit -f and separately save the body and status code. After corruption, we re-compare the SHA-256 to ensure that the failed write request did not initialize the data.
In the actual verification, the service was active and the port was also LISTEN, but the request returned 503. Restart=on-failure is a policy for when a process exits with failure, so it does not automatically fix incorrect data in a living process. This is why we need to check the process, socket, HTTP response, and data meaning individually.
Restore the Real Data Path and Verify New Writes
After stopping the service, retain the corrupted files as evidence and restore only one data file from the verified archive to its original location. Do not indiscriminately overwrite normal programs and service units. Since restoration occurs on the same VM here, we can directly compare previously recorded numeric UID and GID. If moving to a different server, you need to reset them based on the target account, as in service migration and ownership mapping .
sudo bash <<'ROOT'
set -euo pipefail
dest=/var/backups/lf-capstone36
systemctl stop lf-capstone36.service
cp -a /var/lib/lf-capstone36/state.json "$dest/broken-state.json"
cd "$dest"
sha256sum -c service.sha256
tar --acls --xattrs --numeric-owner --same-owner -xpf service.tar -C / var/lib/lf-capstone36/state.json
sha256sum -c state-before.sha256
test "$(stat -c '%u:%g:%a' /var/lib/lf-capstone36/state.json)" = "$(cat state-mode.txt)"
systemctl start lf-capstone36.service
sleep .3
test "$(curl -fsS http://127.0.0.1:8086/)" = '{"count":2}'
test "$(curl -fsS -X POST http://127.0.0.1:8086/increment)" = '{"count":3}'
systemctl restart lf-capstone36.service
sleep .3
test "$(curl -fsS http://127.0.0.1:8086/)" = '{"count":3}'
test "$(curl -fsS -X POST http://127.0.0.1:8086/increment)" = '{"count":4}'
test "$(curl -fsS http://127.0.0.1:8086/)" = '{"count":4}'
stat -c '%U:%G:%a' /var/lib/lf-capstone36/state.json
echo ACTUAL_RESTORE_COUNT2_NEW_WRITE3_RESTART3_NEW_WRITE4_PASSED
ROOT
The stored state hash describes the backup value 2. Compare it immediately after restoration, before the next POST. Successful increments to 3 and 4 deliberately change the file and its hash; record the new expected value instead of treating every later hash difference as corruption.
The expected result is OK for the archive and data hash, and lf-capstone36:lf-capstone36:600. After reading the restored value 2, record 3 via POST, restart the process, read 3 again, and verify writing once more to get 4. Distinguish between the fact that the file was extracted and the fact that read/write operations are possible after restoration.
Interpret Failed Checks Before Making Changes
| Symptoms | Points to Check First |
|---|---|
| Service Start Failure | Check the executable file, user, path errors in journalctl, and the result of systemd-analyze verify. |
| Process Active, No Port | Check the MainPID, execution arguments, bind address, duplicate ports, and logs right after start. |
| Read 503 | Check file existence, read permissions, JSON syntax, and integer types, and preserve corrupted files first. |
| Read success, write 503 | Check data directory permissions, ReadWritePaths, remaining space, inode, and file replacement conditions. |
| Values change after restart | Check actual data path, mount, whether saving was successful, and other write processes. |
If the restoration is complete, record service status, ports, responses, data hashes, and normal values changed after restoration in the work log. If there were normal transactions or uploads after the restoration point, restoring the old backup over current data could lose those newer changes, so decide first on stopping writes, separate preservation, and reapplication methods.
Clean Up the Lab Service and Retain Its Backup
The block below only cleans up services, code, data, and accounts newly created in this article. Keep backups and evidence of damage in a root-only directory for comparison. When deleting copies after the retention period, select the exact files and do not delete backups of other services together.
sudo bash <<'ROOT'
set -euo pipefail
systemctl disable --now lf-capstone36.service
rm -- /etc/systemd/system/lf-capstone36.service /opt/lf-capstone36/app.py /var/lib/lf-capstone36/state.json
rmdir /opt/lf-capstone36 /var/lib/lf-capstone36
systemctl daemon-reload
userdel lf-capstone36
test -z "$(ss -H -ltn 'sport = :8086')"
echo OWN_SAMPLE_REMOVED_VALID_BACKUP_AND_BROKEN_EVIDENCE_RETAINED
ROOT




