Debian 13 Tutorial / Use journalctl and logrotate for Log Inspection and Retention
Debian 13 manages logs by distinguishing between journals viewed with journalctl and file logs that applications write directly. You first export the records necessary for troubleshooting, and for file logs, you need to set rotation, compression, and retention counts according to the program's reopening method. This tutorial demonstrates the operation of logrotate with dedicated files and services without deleting operational logs.
Distinguish the Journal from Application File Logs
systemd-journald collects service standard output and error, and system messages into journals with fields. journalctl is a tool to view the stored records. Conversely, application files like /var/log/nginx can be opened and written by the program itself, and logrotate usually renames, compresses, and retains them.
| Target | Main tools | Items to check |
|---|---|---|
| systemd journal | journalctl and journald settings | Service, boot, time, priority, storage location, and size policy |
| Program file logs | Application settings and logrotate | File path, ownership and open files, reopening on rotation and timing of compression |
Do not delete journal files with rm like regular text files or include them in logrotate. If file logs are deleted but space remains, it is also related to deleted files that are still open. Use the disk-space and deleted-open-file diagnosis lab to compare df and du, and first identify the services that write logs.
Filter by Service, Boot and Time
Determine the service and time when the problem occurred and query as follows. Replace the time example with the actual server failure time. Output may include information such as accounts, addresses, and tokens, so review before sharing externally.
sudo journalctl --list-boots --no-pager
sudo journalctl -b -u nginx.service -n 50 --no-pager
sudo journalctl -b -u nginx.service --since '2026-11-16 22:00:00' --until '2026-11-16 23:00:00' -o short-iso --no-pager
sudo journalctl -b -p warning --no-pager
sudo journalctl -b -k --no-pager
-b selects the current boot, -b -1 selects the previously saved boot. First check existence with --list-boots. -u is for services, -k is for kernel messages, -p warning narrows the scope to warnings and more severe priorities. If you specify the error level, you may miss clues at the info level, so don’t look at only that from the beginning.
The practice VM had previous boot journals preserved and journalctl --disk-usage reported 40MiB combining active and archived files. This is the observation in the current example and not the default retention for all Debian servers. Query permissions and filters can be checked in Debian journalctl manual.
For a user service, use journalctl --user -u UNIT, replacing UNIT with its actual service name. If system-journal access is restricted, use sudo for the necessary query rather than adding the account to a broadly privileged group for convenience.
Create the Complete Dedicated Log-writing Service
This is a dedicated practice environment for Debian 13, systemd 257, Python 3 and logrotate 3.22. The example below assumes a VM with a general user lfstudent. If using a different account, change the unit's User and Group and logrotate create owner to the same existing account. You can check if the account exists with id lfstudent.
Prepare after installation and path verification. If a dedicated name already exists, stop without overwriting existing files. This file only records practice data and is not linked to existing Nginx logs.
sudo apt install logrotate python3
id lfstudent
for p in /var/log/lf-log29 /var/backups/lf-debian13-29 /usr/local/libexec/lf-log29-writer.py /etc/systemd/system/lf-log29.service; do
if test -e "$p"; then echo "Already exists: $p" >&2; exit 1; fi
done
sudo install -d -m 755 /var/log/lf-log29 /usr/local/libexec
sudo install -d -m 700 /var/backups/lf-debian13-29
sudo install -m 640 -o lfstudent -g lfstudent /dev/null /var/log/lf-log29/app.log
Save the entire following Python program. It keeps the file open while writing. On SIGHUP (the hang-up signal), it requests a close and reopen of the same path on its next iteration. A practice limitation of recording for up to approximately 10 minutes has been applied. Not all general services support SIGHUP re-opening.
sudo tee /usr/local/libexec/lf-log29-writer.py > /dev/null <<'PY'
import os,signal,time
path='/var/log/lf-log29/app.log'
reopen=True
running=True
def hup(signum,frame):
global reopen
reopen=True
def stop(signum,frame):
global running
running=False
signal.signal(signal.SIGHUP,hup)
signal.signal(signal.SIGTERM,stop)
f=None
try:
for number in range(6000):
if not running: break
if reopen:
if f is not None: f.close()
f=open(path,'a',buffering=1)
reopen=False
print(f'<6>opened inode={os.fstat(f.fileno()).st_ino}',flush=True)
f.write(f'line={number:04d} sample log record\n')
time.sleep(.1)
finally:
if f is not None: f.close()
print('<6>writer stopped',flush=True)
PY
Confirm id lfstudent succeeds before preparing files; stop if the account or any installation step fails. After saving the unit, check it with sudo systemd-analyze verify /etc/systemd/system/lf-log29.service before daemon-reload and start.
Save the entire following unit. Standard output records the inode of the open file, and the file logs numbered entries. File reopening is requested by the HUP sent through ExecReload. If applied as-is to a real program that does not support this, it may terminate, so the program documentation should be checked.
sudo tee /etc/systemd/system/lf-log29.service > /dev/null <<'UNIT'
[Unit]
Description=LF log rotation lab 29
[Service]
Type=exec
User=lfstudent
Group=lfstudent
ExecStart=/usr/bin/python3 /usr/local/libexec/lf-log29-writer.py
ExecReload=/bin/kill -HUP $MAINPID
SyslogIdentifier=lf-log29
UNIT
sudo chmod 644 /usr/local/libexec/lf-log29-writer.py
sudo systemctl daemon-reload
sudo systemctl start lf-log29.service
sleep 1
sudo systemctl is-active lf-log29.service
sudo tail /var/log/lf-log29/app.log
sudo journalctl -b -u lf-log29.service -n 10 --no-pager
In the file, lines like line=0000 sample log record increase, and in the journal, you can see opened inode=number. If the program was terminated during practice, check the logs and status, and then restart only the dedicated service. Distinguish between normal termination when reaching the maximum execution time of the example and permission errors.
Create an Independent logrotate Configuration and Check Debug Mode
The practice settings are placed as independent files in the backup folder, not in /etc/logrotate.d. Therefore, the system's daily timer does not automatically handle this example, and only manual verification is performed. Using a separate --state also does not touch the system default state file.
sudo tee /var/backups/lf-debian13-29/logrotate.conf > /dev/null <<'CONF'
/var/log/lf-log29/app.log {
size 100
rotate 2
compress
delaycompress
missingok
notifempty
create 0640 lfstudent lfstudent
sharedscripts
postrotate
/usr/bin/systemctl reload lf-log29.service
endscript
}
CONF
sudo chmod 600 /var/backups/lf-debian13-29/logrotate.conf
sudo logrotate --debug --state /var/backups/lf-debian13-29/logrotate.state /var/backups/lf-debian13-29/logrotate.conf
sudo ls -l /var/log/lf-log29
--debug shows the reasoning but does not change logs or state files. In execution verification, the writing process was temporarily paused to compare hashes, confirming that the file contents, rotated files, and separate state files do not change. Readers do not need to stop the operational process for this verification. The fact that the hash of the file being written changed does not mean debug modified the file.
| Directives | Meaning and selection criteria |
|---|---|
| size 100 | This is a small practice limit based on 100 bytes. It is not a real-time rotation monitoring function at exactly this size. |
| rotate 2 | Maintain two rotated backups. The setting does not guarantee two days. |
| compress and delaycompress | Compress, but defer compression of the most recent archive until the next rotation. |
| create 0640 | Specify the permissions and ownership of the new active log. |
| postrotate | In this example, request reload so that the service opens a new file after rotation. |
The Debian logrotate manual describes supported options and debug/state behavior. Check precedence when combining conditions such as size and daily. A small size threshold does not prevent substantial growth between invocations when logrotate runs only once a day.
Verify Rotation, Compression and File Reopening
The following is the first rotation with --force applied only to the dedicated configuration. It is a forced execution to clarify conditions when testing initially and is not used for the entire production configuration.
sudo logrotate --verbose --force --state /var/backups/lf-debian13-29/logrotate.state /var/backups/lf-debian13-29/logrotate.conf
sleep 1
sudo ls -li /var/log/lf-log29
sudo tail /var/log/lf-log29/app.log
sudo journalctl -b -u lf-log29.service -n 15 --no-pager
During the first rotation, the existing file becomes app.log.1 and the new app.log has a different inode. This program opens the new path in the next iteration after receiving HUP. In that short interval, lines written to the previous file may remain in the previous archive. What is important is whether subsequent records continue to go into the new active file. In verification, we even compared whether the open file descriptor inode in /proc matched the active file inode.
Run the following block twice, giving three rotations in total. After each rotation, wait briefly to verify that content has appeared in the active file. If the new file is empty, it may skip rotation due to notifempty, and if the program has exited, postrotate reload may fail.
sudo logrotate --verbose --force --state /var/backups/lf-debian13-29/logrotate.state /var/backups/lf-debian13-29/logrotate.conf
sleep 1
sudo ls -li /var/log/lf-log29
sudo gzip -t /var/log/lf-log29/app.log.2.gz
sudo gzip -cd /var/log/lf-log29/app.log.2.gz | head
| Number of rotations | Active log | Archives |
|---|---|---|
| First | Resume writing to app.log | app.log.1, not yet compressed |
| Second | Recorded in new app.log | app.log.1 and app.log.2.gz |
| Third | Recorded in new app.log | Keep two backup versions, remove the oldest first backup content |
In the actual three rotations, the gzip check passed and app.log.3.gz did not remain. Inode numbers can be reused, so do not expect them to increase globally. If you just rename without reopening, the process can continue to write to the previous inode. Copytruncate is not used as a general solution for all programs because there is a possibility of losing records between copying and truncating.
Export the Journal and Preserve File Logs Before Cleanup
When preserving logs for investigation, record filters, time zones, and boot conditions together. The text is easy to read, and the export format can be used when keeping journal fields together. The fact that an export file was created does not mean it was re-registered in the main journal.
sudo systemctl stop lf-log29.service
sudo journalctl --sync
sudo journalctl -b -u lf-log29.service -o short-iso --no-pager | sudo tee /var/backups/lf-debian13-29/service-journal.txt > /dev/null
sudo journalctl -b -u lf-log29.service -o export --no-pager | sudo tee /var/backups/lf-debian13-29/service-journal.export > /dev/null
sudo cp -a /var/log/lf-log29/app.log /var/log/lf-log29/app.log.1 /var/log/lf-log29/app.log.2.gz /var/backups/lf-debian13-29/
sudo ls -lh /var/backups/lf-debian13-29
In practice, reopening records and normal shutdown messages were included in the exported journal, and file logs and compressed backups were also copied separately. If there are files skipped due to intermediate rotations, preserve according to the actual list, and do not treat the backup as complete while ignoring copy failures.
Review Retention Policy and Remove the Lab
The journald storage policy is checked with the merge settings as follows. Storage=auto changes the storage method depending on the presence of the permanent journal directory, etc. To preserve previous boots, also check whether it is actually stored permanently and whether it is writable.
systemd-analyze cat-config systemd/journald.conf
sudo journalctl --disk-usage
systemctl cat logrotate.timer
systemctl list-timers logrotate.timer --no-pager
SystemMaxUse and RuntimeMaxUse in journald are the usage policies for the permanent and runtime storage respectively, and MaxRetentionSec is an item that limits the retention period of old records. Data may be removed earlier due to capacity limits, so it does not guarantee a minimum retention period. Based on Debian journald retention configuration manual, you decide together on disk space availability, log generation volume, and required retention period.
When actually changing the policy, back up the existing settings and specify the [Journal] section in a separate .conf file managed by yourself in /etc/systemd/journald.conf.d, then check the merged result. Apply by running sudo systemctl restart systemd-journald and check the service status and log errors. When reverting, only remove the drop-in you created and apply it again. Logs that have already been deleted according to the retention policy are not restored by reverting the settings, so first preserve any necessary records externally.
This practice did not change the existing journald retention policy or delete the operational journal using vacuum. The vacuum function of journalctl is a deletion operation and does not guarantee that usage, including active files, will immediately drop to the desired level. Logs are not deleted in bulk before investigation just because storage is insufficient.
Stop the practice service, check the copy, and then clean up only the dedicated files. Since nothing was added to /etc/logrotate.d, other rules of the system timer remain in place.
sudo rm -- /etc/systemd/system/lf-log29.service /usr/local/libexec/lf-log29-writer.py
sudo systemctl daemon-reload
sudo rm -- /var/log/lf-log29/app.log /var/log/lf-log29/app.log.1 /var/log/lf-log29/app.log.2.gz
sudo rmdir /var/log/lf-log29
After verification, the dedicated service and log path were removed, and evidence along with configuration and state files were preserved in /var/backups/lf-debian13-29. Only rules with appropriate actual size, execution schedule, and reloading commands are added to the operational logrotate, and the 100-byte and forced rotation from this practice are not used as operational defaults.




