Linux Tutorial / Reproduce Failures and Practice Evidence-Based Recovery
Practice recovery in an isolated environment by verifying a working baseline, introducing one fault, and restoring the changed condition using evidence. Validate both service state and the user-visible response before declaring recovery complete.
Prepare VM snapshots and console recovery and compare evidence before and after each change using log-based investigation.
System configuration exercises use a conventional Ubuntu 26.04 LTS or Rocky Linux 9 VM as the reference environment. Containers, WSL, and other releases can differ in service and package configuration. Prepare console recovery and record your versions using the virtual machine setup lesson.
Define the isolated lab and a working baseline
Use an Ubuntu or Rocky Linux lab VM with a systemd user manager, Python 3, and curl. This exercise does not alter production Nginx, SSH, firewall, or disks. Check ss -lnt for a free loopback port 18080 and change all matching lab values if it is already in use.
Confirm the actual Python path with command -v python3. Do not overwrite an existing unit or lab directory with these names. The Python development HTTP server is restricted to this loopback exercise and is not a production Internet server.
mkdir -p "$HOME/linux-recovery-lab/site" "$HOME/.config/systemd/user"
printf "recovery baseline\n" > "$HOME/linux-recovery-lab/site/index.html"
cat > "$HOME/.config/systemd/user/linux-course-recovery.service" <<'EOF'
[Unit]
Description=Isolated Linux recovery lab
[Service]
Type=simple
WorkingDirectory=%h/linux-recovery-lab/site
ExecStart=/usr/bin/python3 -m http.server 18080 --bind 127.0.0.1
EOF
systemctl --user daemon-reload
systemctl --user start linux-course-recovery.service
systemctl --user is-active linux-course-recovery.service
curl --fail --max-time 5 http://127.0.0.1:18080/
cp "$HOME/.config/systemd/user/linux-course-recovery.service" "$HOME/linux-recovery-lab/service.good"Introduce a fault only after seeing active and the expected page. The good unit has been saved and the service has not been enabled at boot. If the baseline fails, resolve the Python path, unit configuration, or port conflict before continuing.
Experiment 1: break only the executable path
sed -i 's@ExecStart=/usr/bin/python3 @ExecStart=/nonexistent/course-python @' "$HOME/.config/systemd/user/linux-course-recovery.service"
systemctl --user daemon-reload
systemctl --user restart linux-course-recovery.service
systemctl --user status linux-course-recovery.service --no-pager
journalctl --user -u linux-course-recovery.service --since "5 minutes ago" --no-pagerThe unit now points to a nonexistent executable, so the service cannot stay active. Depending on timing, the start command may fail immediately or the subsequent status may reveal failure. Logs should identify an execution failure; systemd may report 203/EXEC. Record the actual message instead of treating every failed service as a network problem.
Restore configuration and verify the response
cp "$HOME/linux-recovery-lab/service.good" "$HOME/.config/systemd/user/linux-course-recovery.service"
systemctl --user daemon-reload
systemctl --user reset-failed linux-course-recovery.service
systemctl --user start linux-course-recovery.service
systemctl --user is-active linux-course-recovery.service
curl --fail --max-time 5 http://127.0.0.1:18080/Restore the known-good file, reload the manager, and start the service. Reset-failed can clear failure state and start-limit counters, but does not repair an invalid configuration. Recovery is complete only when service state and the expected page both return.
Experiment 2: distinguish a closed port from an HTTP error
curl -i --max-time 5 http://127.0.0.1:18080/missing-resource
systemctl --user stop linux-course-recovery.service
curl -i --connect-timeout 2 --max-time 5 http://127.0.0.1:18080/
ss -lnt
systemctl --user start linux-course-recovery.service
curl --fail --max-time 5 http://127.0.0.1:18080/The first request receives HTTP 404 from a running server. After stopping the service, the next request fails at connection level, typically with refusal, rather than receiving HTTP 404. Compare curl behavior with and without --fail, since an ordinary 404 transfer may exit successfully. This distinguishes service, transport, and HTTP failures without changing a firewall.
Keep a one-change recovery record
| Record | Example evidence |
|---|---|
| Baseline | Active unit, listener, expected page |
| Change | Only ExecStart or service stop |
| Evidence | Time, exit status, journal, HTTP, socket |
| Recovery | Known-good configuration or service start |
| Retest | Same URL, port, and expected body |
| Prevention | Pre-deployment validation and health checks |
Do not fill the system disk, block management SSH, or delete random files in this exercise. More disruptive tests require a separate clone, bounded storage, verified console access, and backups. The objective is evidence-based diagnosis and recovery, not maximum disruption.
Stop the lab and remove only its own unit
systemctl --user stop linux-course-recovery.service
systemctl --user reset-failed linux-course-recovery.service
rm -- "$HOME/.config/systemd/user/linux-course-recovery.service"
systemctl --user daemon-reload
ss -lntConfirm the loopback listener has disappeared. Review and remove the lab directory separately; it contains the baseline configuration and static page. In environments with other services, scope failure-state operations to the relevant unit rather than clearing unrelated user failures.
