Debian 13 Tutorial / Diagnose Service Failures, OOM Kills and Disk-related HTTP Errors

Service failures in Debian 13 can be narrowed down by separately checking the cause of process termination, ports and actual responses, and memory and file system status. The measures differ in cases such as exit code 42, out-of-memory (OOM) termination due to memory limits, and when the process is alive but returns HTTP 503 due to lack of space. This tutorial reproduces the three situations under small resource limits, records the evidence, and then removes the cause.

Capture Evidence Before Restarting

First, determine the time of the problem, the failed request, and the target path. If you restart unconditionally, the symptoms may temporarily disappear, but the cause may remain and you can miss the termination status and resource information. Replace the following nginx.service with the actual service name to be investigated.

date --iso-8601=seconds
sudo systemctl status nginx.service --no-pager
sudo systemctl show nginx.service -p ActiveState -p SubState -p Result -p ExecMainCode -p ExecMainStatus -p MainPID
sudo journalctl -b -u nginx.service -n 80 --no-pager
ss -ltn
free -h
vmstat -y 1 3
df -hT /
df -i /

status or is-active can return a non-zero exit code in a failure state. In the collection script, branch to record that state, and do not skip later checks unconditionally. Since logs may contain addresses, accounts, and request information, restrict storage paths and sharing scope.

Observation Additional checks Statements to avoid
service failed Result, exit code, signal and recent log Not all failures are due to out of memory.
Active but request failure Listener, HTTP status and data reads/writes Active does not mean user functionality is normal.
Memory seems to be low Available memory, swap changes, service limits and OOM evidence OOM should not be diagnosed by only one free column.
Write Failure Mount, block, inode, and permission of the failed path Do not just check the free space of the root disk and finish.

Prepare the Isolated Environment and Complete Program

A control group (cgroup) groups processes for resource control. tmpfs is a memory-backed filesystem with a separately configurable size limit. Install missing tools with sudo apt install python3 curl procps. Stop if lfstudent is absent, port 8080 is occupied, or any prerequisite check fails; the collision guard exits its shell, so use a dedicated lab shell.

A dedicated Debian 13 VM with sudo, Python 3, curl, and the memory controller of cgroup v2 is required. The verification VM had about 4GiB of memory, but the practice was conducted only within a 64MiB limit for a single service and a new 4MiB tmpfs. It does not deplete actual operating services or the root disk.

This is an example of an environment with a regular user lfstudent. For other accounts, adjust the User, Group, and tmpfs owner UID/GID to actual values. First, check that the name and port are empty.

id lfstudent
stat -fc %T /sys/fs/cgroup
cat /sys/fs/cgroup/cgroup.controllers
ss -ltn
for p in /usr/local/libexec/lf-diag31.py /etc/systemd/system/lf-diag31.service /mnt/lf-diag31 /var/backups/lf-debian13-31; do
    if test -e "$p"; then echo "Already exists: $p" >&2; exit 1; fi
done
sudo install -d -m 755 /usr/local/libexec
sudo install -d -m 700 /var/backups/lf-debian13-31

The file system should be cgroup2fs, and the controller list should include memory. The following full program is divided into exit42, oom, and serve modes. The oom mode allocates at most 128MiB of bytearray payload, in addition to interpreter overhead, and must be executed with the subsequent cgroup limit command. GET in serve is an experimental confirmation request that writes 4KiB to the isolated path and is not used as a general web application implementation example.

sudo tee /usr/local/libexec/lf-diag31.py > /dev/null <<'PY'
import os,sys
from http.server import BaseHTTPRequestHandler,HTTPServer
mode=sys.argv[1]
if mode=='exit42':
    print('intentional exit code 42',flush=True)
    raise SystemExit(42)
if mode=='oom':
    chunks=[]
    for i in range(8):
        chunks.append(bytearray(16*1024*1024))
        print(f'allocated_chunks={i+1}',flush=True)
    raise SystemExit('Expected cgroup memory limit')
if mode!='serve': raise SystemExit('Unknown mode')
class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        try:
            with open('/mnt/lf-diag31/probe.bin','ab') as f:
                f.write(b'.'*4096)
                f.flush()
                os.fsync(f.fileno())
            status=200;body=b'write probe ok\n'
        except OSError as e:
            print(f'write probe errno={e.errno}: {e.strerror}',flush=True)
            status=503;body=b'write probe failed\n'
        self.send_response(status)
        self.send_header('Content-Type','text/plain')
        self.send_header('Content-Length',str(len(body)))
        self.end_headers();self.wfile.write(body)
HTTPServer(('127.0.0.1',8080),Handler).serve_forever()
PY
sudo chmod 644 /usr/local/libexec/lf-diag31.py

Distinguish Exit Codes from OOM Kills

The first test intentionally returns exit code 42. As --wait propagates this result, a command failure is expected. Use an interactive lab shell without automatic exit on failure for these deliberately failing systemd-run commands, or explicitly capture their nonzero status before continuing to the diagnostic queries.

sudo systemd-run --unit=lf-exit31 --wait --property=User=lfstudent --property=RuntimeMaxSec=10s /usr/bin/python3 /usr/local/libexec/lf-diag31.py exit42
sudo systemctl show lf-exit31.service -p Result -p ExecMainCode -p ExecMainStatus
sudo journalctl -b -u lf-exit31.service --no-pager

The actual result was Result=exit-code, ExecMainCode=1, ExecMainStatus=42. The 1 in ExecMainCode is the code of the termination method and does not mean the program exited with 1. Read the status code=exited together with the actual exit status 42.

Next, for a single service, we specify MemoryMax=64M, MemorySwapMax=0, and a 10-second execution limit. The limit is applied only to a dedicated transient unit without affecting other jobs of the account. Stop if these limits cannot be applied; never run the oom mode outside its bounded unit.

sudo systemd-run --unit=lf-oom31 --wait --property=User=lfstudent --property=MemoryMax=64M --property=MemorySwapMax=0 --property=RuntimeMaxSec=10s --property=OOMPolicy=stop /usr/bin/python3 /usr/local/libexec/lf-diag31.py oom
sudo systemctl show lf-oom31.service -p Result -p ExecMainCode -p ExecMainStatus -p MemoryMax -p MemorySwapMax -p MemoryPeak
sudo journalctl -b -u lf-oom31.service --no-pager

OOM (Out of Memory) refers to a situation where memory cannot be allocated. cgroup is a kernel feature that applies resource limits to groups of processes. Even if the host has free memory, the process may be terminated due to the cgroup limit. In verification, Result=oom-kill, ExecMainCode=2, ExecMainStatus=9 and MemoryMax and MemoryPeak=67,108,864 bytes, MemorySwapMax=0 were observed.

You cannot conclude OOM with signal 9 alone. A forced stop by the administrator could also generate the same signal, so we look at the Result along with the journal at that time. We also investigate kernel or systemd-oomd intervention depending on the situation. The meaning of limits per service can be found in systemd resource control description, and hierarchy and memory events can be checked in Linux cgroup v2 documentation.

Do not immediately change the limit to unlimited. Adjust it after checking normal workload requirements, leaks, concurrent jobs, and overall host budget. Removing the limit is different from solving the root cause.

Diagnose a Running Service That Cannot Write

Create a separate 4MiB tmpfs and loopback service. The lfstudent UID/GID of the verification VM was 1000, and below we get the actual values using id. You should have checked earlier that the existing path and 8080 are free.

sudo mkdir /mnt/lf-diag31
sudo mount -t tmpfs -o "size=4M,nr_inodes=64,uid=$(id -u lfstudent),gid=$(id -g lfstudent),mode=0700,nosuid,nodev" tmpfs /mnt/lf-diag31
findmnt --mountpoint /mnt/lf-diag31
sudo tee /etc/systemd/system/lf-diag31.service > /dev/null <<'UNIT'
[Unit]
Description=LF bounded write failure HTTP lab 31
[Service]
Type=exec
User=lfstudent
Group=lfstudent
ExecStart=/usr/bin/python3 /usr/local/libexec/lf-diag31.py serve
UNIT
sudo systemctl daemon-reload
sudo systemctl start lf-diag31.service
curl --noproxy '*' -fsS http://127.0.0.1:8080/

The normal response is write probe ok. If requests fail immediately after starting, check status and logs and retry after a while. Confirm the successful baseline response before proceeding to the next step.

Below, as lfstudent, we attempt to write 1MiB up to 8 times. The 4MiB mount fills up first, causing ENOSPC (no space left on device). Do not run it if you cannot verify the mount of that tmpfs.

sudo -u lfstudent python3 - <<'PY'
import errno,pathlib
p=pathlib.Path('/mnt/lf-diag31/fill.bin')
try:
    with p.open('xb') as f:
        for i in range(8):
            f.write(b'x'*1048576);f.flush()
except OSError as e:
    if e.errno!=errno.ENOSPC:raise
    print('BOUNDED_TMPFS_ENOSPC')
else:raise SystemExit('Expected bounded ENOSPC')
PY
curl --noproxy '*' -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/
sudo systemctl is-active lf-diag31.service
ss -ltn | grep '127.0.0.1:8080'
findmnt --target /mnt/lf-diag31 -o TARGET,SOURCE,FSTYPE,OPTIONS
df -hT /mnt/lf-diag31
df -i /mnt/lf-diag31
sudo journalctl -b -u lf-diag31.service -n 20 --no-pager

In reality, HTTP returned 503, but the service was active, and the port was LISTEN. The tmpfs block used the entire 4MiB, and only 3 out of 64 inodes were used. Writing failures were linked to journal error number 28: No space left on device. The cause was not service downtime or inode exhaustion, but rather block exhaustion on the dedicated tmpfs.

Now, remove only the known lab filler file and make the same request to the service again. Do not restart the service.

sudo rm -- /mnt/lf-diag31/fill.bin
curl --noproxy '*' -fsS http://127.0.0.1:8080/
df -hT /mnt/lf-diag31

In verification, the response returned to write probe ok. In operations, cleanup is performed only when the target that can be deleted is clear. Regular logs or data are not deleted indiscriminately; we evaluate the producer, retention, rotation, and expansion plans. If it is a permission error or read-only mount, freeing space will not resolve the issue, so we branch by the error message.

Interpret Memory and I/O Metrics Within Their Sampling Period

In free, free reports unused memory, while available estimates memory that new applications could use without swapping. Cached memory is not necessarily wasted. The free manual explains how each column is calculated.

free -h
vmstat -y 1 3
ps -eo pid,comm,rss,%mem,%cpu --sort=-rss | head

The initial default report of vmstat may include averages since boot, so we omit it with -y and observe three times at 1-second intervals. si and so are swap input/output, r indicates runnable tasks, and b points to tasks blocked waiting for I/O completion, providing clues. You cannot determine disk failure or CPU bottleneck from a single sample of wa or CPU values. Check the sample range in the vmstat manual and compare it with the time of the incident.

After verification, this VM had approximately 3.4 GiB available and 0 swap usage, but this does not represent the free space or operational capacity at all times. In reports, we separate observed values, judgments supported by those values, and unverified assumptions.

Preserve Evidence, Confirm Recovery and Clean Up

Retain evidence before clearing a failed state. reset-failed clears the failure indicator but does not fix memory or file issues.

sudo journalctl -b -u lf-oom31.service --no-pager | sudo tee /var/backups/lf-debian13-31/oom-journal.txt > /dev/null
sudo journalctl -b -u lf-diag31.service --no-pager | sudo tee /var/backups/lf-debian13-31/write-failure-journal.txt > /dev/null
sudo systemctl show lf-oom31.service -p Result -p ExecMainCode -p ExecMainStatus -p MemoryMax -p MemoryPeak | sudo tee /var/backups/lf-debian13-31/oom-state.txt

Exit 42 indicates an intended program termination, oom-kill signifies termination related to actual service memory limits, and 503 was a write failure while in active state. Since we rechecked after freeing space until response, the last case confirms functionality recovery.

sudo systemctl stop lf-diag31.service
sudo rm -- /mnt/lf-diag31/probe.bin
sudo umount /mnt/lf-diag31
sudo rmdir /mnt/lf-diag31
sudo rm -- /usr/local/libexec/lf-diag31.py /etc/systemd/system/lf-diag31.service
sudo systemctl daemon-reload
sudo systemctl reset-failed lf-exit31.service lf-oom31.service
sudo systemctl --failed --no-pager

In verification, dedicated services, mounts, and files were removed, and the failed service list was empty. In actual incidents, we finalize by collecting evidence, taking actions appropriate to the cause, confirming responses and data, and recording the observation range. We do not consider user functionality and data as recovered just because the process has restarted.