Debian 13 Tutorial / Diagnose Disk Space, Inodes, Deleted-open Files and tmpfs

When you cannot write files on Debian 13, you should first separately check the block capacity and inodes of the path in question. If space does not decrease even after deleting files, check for open deleted files, and if it is a /tmp issue, check the tmpfs limit, which is separate from the root disk. In this tutorial, we compare three causes in a small dedicated practice space without deleting operational files.

Start with the Path That Failed

An inode is an object in the file system that manages information such as file type, permissions, ownership, and data location. Even empty files use inodes, so you may not be able to create new files even if there is plenty of byte-level space. Conversely, a few large files can exhaust the blocks.

The following command should be executed by replacing it with the path where writing actually failed. It is important not to judge /var/tmp and /tmp together as a root disk problem.

findmnt --target /var/tmp -o TARGET,SOURCE,FSTYPE,OPTIONS
df -hT /var/tmp
df -i /var/tmp
sudo du -x -h --max-depth=1 /var | sort -h

findmnt shows the actual mount containing the path, and df shows the space of that file system. df -i checks inode availability. du sums the files linked under the accessible directory tree, and -x prevents crossing into other file systems. If permission errors occur, the total is incomplete. Do not hide those errors and then treat the total as accurate. Running du on a large tree takes time and performs I/O, so narrow the range from the problem path.

Observation results Causes to check first Action determination
Almost no block space left, but inodes remain Large files, logs, actual data increase Check the owning service and retention policy, and plan for moving, rotating, or expanding.
Blocks remain, but no inodes A very large number of small files, cache or queue increase Identify the problematic directory and cause of creation, then use the application's cleanup function.
df shows high usage, but du total is small Open deleted files, files hidden by mounts, permission or aggregation scope differences Compare with the same file system and permissions, then investigate the process and mount structure.
/ has available space, but writing to /tmp fails Size and inode limits of a separate tmpfs or memory status Check df and mount options of /tmp along with free.

Permission denied, Read-only file system, Disk quota exceeded are clues that differ from lack of space. Even if there is available space, failure can occur due to ownership, access control lists (ACLs), read-only mounts, or quotas. Because of reserved space and metadata in ext4, df and the file content size do not need to match exactly. Space discrepancies alone do not determine corruption.

Compare Block and Inode Exhaustion on a 4 MiB tmpfs

The lab uses Python 3, e2fsprogs (mkfs.ext4 and e2fsck), util-linux (mount, findmnt and losetup), and standard GNU file tools. Check command -v python3 mkfs.ext4 e2fsck mount findmnt losetup truncate df du first. If something is missing, inspect the package changes before installing the needed Debian package. Run the setup in a dedicated Bash session: an unexpected failure under set -euo pipefail can end that shell. Reconnect if needed, inspect the lab mount and files, then use the targeted cleanup rather than recreating resources over them.

A separate Debian 13 practice VM, a regular user who can use sudo, and Python 3 are required. The example below creates a dedicated tmpfs of up to 4MiB and 32 inodes without filling /tmp. tmpfs uses virtual memory, and if swap is enabled, some pages may go to swap. Do not assume it is stored only in physical RAM, and contents are lost when unmounted.

The next block should be executed in the same Bash terminal. If the path already exists, it stops and does not delete the existing path before continuing. In this VM, the regular user UID and GID were both 1000, but the commands use the current account values.

set -euo pipefail
tm=/mnt/lf-space24-tmpfs
if test -e "$tm"; then
    echo "Path already exists: $tm" >&2
    exit 1
fi
sudo mkdir "$tm"
sudo mount -t tmpfs -o "size=4M,nr_inodes=32,uid=$(id -u),gid=$(id -g),mode=0700,nosuid,nodev" tmpfs "$tm"
findmnt --mountpoint "$tm"

After verifying that the mount result is tmpfs and the size corresponds to 4MiB, run the following full example as a regular user. Write attempts are limited to a maximum of 8MiB, and the number of empty files is limited to a maximum of 40. Only the expected ENOSPC (no space left on device) error is handled, and other errors will stop the process.

python3 - <<'PY'
import errno, pathlib, subprocess
p=pathlib.Path('/mnt/lf-space24-tmpfs')
try:
    with (p/'fill.bin').open('xb') as f:
        for i in range(8):
            f.write(b'x' * 1048576)
            f.flush()
except OSError as e:
    if e.errno != errno.ENOSPC: raise
    print('BLOCK_LIMIT_ENOSPC', flush=True)
else:
    raise RuntimeError('Expected bounded block exhaustion')
subprocess.run(['df','-hT',str(p)],check=True)
subprocess.run(['df','-i',str(p)],check=True)
(p/'fill.bin').unlink()
created=[]
try:
    for i in range(40):
        f=p/f'empty-{i}'
        f.touch(exist_ok=False)
        created.append(f)
except OSError as e:
    if e.errno != errno.ENOSPC: raise
    print('INODE_LIMIT_ENOSPC',len(created),flush=True)
else:
    raise RuntimeError('Expected bounded inode exhaustion')
subprocess.run(['df','-hT',str(p)],check=True)
subprocess.run(['df','-i',str(p)],check=True)
for f in created: f.unlink()
PY

In the actual kernel 6.12.107 and Python 3.13.5 environment, the following mutually opposite results were obtained. The 32 inodes include the top-level directory of the mount, so failure occurred after 31 empty files were created.

Experiment Block usage Inode usage
Writing to fill.bin 4.0MiB / 4.0MiB, free 0 2 / 32, free 30
Delete fill.bin, then create empty files Used 0, Free 4.0MiB 32 / 32, Free 0

The file created will be deleted when the example ends. If it was interrupted by another error in the middle, first check the mount target and the files inside. This tmpfs is for disposable practice data only, so you can unmount it with the following command outside the working directory. If you get an error saying it's in use, move every relevant shell’s current directory outside this path and close any file handles held by lab processes before trying again.

cd "$HOME"
sudo umount /mnt/lf-space24-tmpfs
sudo rmdir /mnt/lf-space24-tmpfs

Observe Space Held by a Deleted-open File

Deleting a file name with the unlink operation and closing an opened file are different. Even if the last name in a directory is gone, if a process still has it open, the file's space can be retained. If you always delete logs with rm, you could create this situation and lose the names you intended to investigate.

The following is a practice to create ext4 inside a new 64MiB regular file. This does not initialize an actual disk device. First, check that there is more than 64MiB free space on the root file system and a little extra working space. If the same path already exists, stop and investigate its usage. Execute the entire preparation block below in the same Bash terminal as well.

set -euo pipefail
base=/var/lib/lf-space24
em=/mnt/lf-space24-ext4
df -hT /var/lib /mnt
for p in "$base" "$em"; do
    if test -e "$p"; then
        echo "Path already exists: $p" >&2
        exit 1
    fi
done
sudo install -d -m 700 "$base"
sudo mkdir "$em"
sudo truncate -s 64M "$base/space.img"
sudo mkfs.ext4 "$base/space.img"
sudo mount -o loop,nosuid,nodev "$base/space.img" "$em"
findmnt --mountpoint "$em"
sudo install -d -o "$(id -u)" -g "$(id -g)" "$em/work"

Files created with truncate are initially sparse files, so the logical size of 64MiB is not immediately fully allocated physically. File system metadata and subsequent practice writes use actual space. The target of mkfs must be the newly created space.img and should not be replaced with a disk name like /dev/sda.

The lost+found of ext4 may not be readable by regular users, so without changing ownership recklessly, only the separate work directory is measured. The following code opens and writes an 8MiB file, deletes its name, and compares before and after closing the file within the same process.

python3 - <<'PY'
import os,pathlib,subprocess
p=pathlib.Path('/mnt/lf-space24-ext4/work')
def used():
    return int(subprocess.check_output(['df','--output=used','-B1',str(p)],text=True).splitlines()[1])
def show(label):
    os.sync()
    print(label,flush=True)
    subprocess.run(['df','-B1',str(p)],check=True)
    subprocess.run(['du','-B1','-s',str(p)],check=True)
baseline=used()
f=(p/'held.bin').open('xb')
for i in range(8): f.write(b'x'*1048576)
f.flush();os.fsync(f.fileno())
show('BEFORE_UNLINK');before=used()
(p/'held.bin').unlink()
print('OPEN_FD',os.readlink(f'/proc/self/fd/{f.fileno()}'),flush=True)
show('UNLINKED_OPEN');held=used()
f.close()
show('AFTER_CLOSE');released=used()
if not (before >= baseline+8388608 and held == before and released < held):
    raise RuntimeError((baseline,before,held,released))
print('DELETED_OPEN_FILE_RELEASE_PASSED',baseline,before,held,released,flush=True)
PY
Time df used bytes du work total bytes
After writing 8MiB file 8,436,736 8,389,632
Name deleted, file still open 8,436,736 1,024
After closing file 48,128 1,024

The above figures are the actual measurements from this practice image and may vary depending on the file system creation options and version. The important result is that after deleting the name, only du decreased while df remained the same, and after closing, the df usage also decreased. The output of /proc/self/fd showed held.bin (deleted). os.sync only synchronizes writes before the observation and does not forcibly close open files.

On production servers, check the log reopening feature of the process that opened the file and the service. If lsof is installed, you can investigate open files with a link count less than 1 using sudo lsof +L1, but in the results, devices, files, and processes should be cross-referenced together. Restarting affects connections and operations, so first review the log rotation/reopening method supported by the service. Do not arbitrarily write to another process’s /proc file descriptor or terminate a process directly.

Check the ext4 Image Only After Unmounting

The above practice did not cause ext4 corruption. There is no need to run e2fsck just because the space is full. The Debian e2fsck manual warns against running it on a mounted file system, and although some read-only runs are possible, the inspection results may not be valid.

In this practice, we check that there are no open processes inside work, release the image, and then run a read-only check with -fn. -f forces a full check and -n is the read-only mode that refuses changes. The target is a single practice image file.

cd "$HOME"
sudo umount /mnt/lf-space24-ext4
sudo losetup -j /var/lib/lf-space24/space.img
sudo e2fsck -fn /var/lib/lf-space24/space.img

The result of losetup -j should be empty. If any connections remain, do not delete the image and check which mount or process is using it. In normal practice, the step-by-step check by e2fsck completes and exits with code 0. If it is not 0, preserve the output and exit code without deleting the file immediately. Checking the operational root is a separate recovery task that requires preparing a console, backup, or rescue boot environment and re-verifying the target UUID while the root is unmounted.

After the check completes normally, only the practice image is deleted. Existing data disks, LVM from previous lectures, and /etc/fstab are not modified.

sudo rm -- /var/lib/lf-space24/space.img
sudo rmdir /var/lib/lf-space24 /mnt/lf-space24-ext4
findmnt --target /tmp -o TARGET,SOURCE,FSTYPE,OPTIONS
df -hT / /tmp /var/tmp

Check Debian 13 Temporary-file Storage and Retention

According to Debian 13’s /tmp change notice, the default /tmp is tmpfs and the default size limit is half of memory. This does not mean that the limit is reserved from the start. Separate /tmp settings in the existing fstab can change the result, so first check the actual mount.

findmnt --target /tmp -o TARGET,SOURCE,FSTYPE,OPTIONS
findmnt --target /var/tmp -o TARGET,SOURCE,FSTYPE,OPTIONS
systemctl cat tmp.mount
systemd-tmpfiles --cat-config | less
systemctl cat systemd-tmpfiles-clean.timer
systemctl list-timers systemd-tmpfiles-clean.timer --no-pager
free -h

In less, you can search with /tmp.conf and exit with q. Since cat-config shows multiple settings, check not only the defaults in /usr/lib/tmpfiles.d but also any entries overwritten by files with the same name in /etc/tmpfiles.d. Do not run systemd-tmpfiles --clean for verification purposes. This command can actually perform deletions.

On the new installation VM for this course, /tmp was a tmpfs with a maximum of about 2GiB, and /var/tmp was included in the root ext4. The applied default rules were /tmp 10 days, /var/tmp 30 days, and the cleanup timer was set to 15 minutes after boot and once a day after the last activation. There is no guarantee that files will be deleted immediately just because the date has come; the file timestamps, exclusion rules, and timing of the timer execution all take effect.

Guidance on Debian Temporary File Cleanup Changes also explains the differences between new installations and upgrades. On systems upgraded from Debian 12 to Debian 13, /etc/tmpfiles.d/tmp.conf may be created to maintain previous retention behavior, so you should not assume the same deletion schedule as for a new installation. /var/tmp generally remains after a reboot but is not a permanent storage location.

For small and short-lived temporary files, use /tmp, and for larger temporary work, choose a separate path with sufficient space. After ensuring that the service follows the TMPDIR setting, change it accordingly, and place persistent data and backups in dedicated state/backup paths. As per systemd's temporary directory usage guidelines , use secure creation methods like mktemp in shared temporary directories. The tmpfs limits and swap behavior can be checked in Linux kernel tmpfs documentation , and the priority of cleanup rules and time determination can be found in Debian tmpfiles.d documentation .