Rocky Linux 10 Tutorial / Query journals and rotate file logs safely
To manage Rocky Linux 10 logs, identify the journal and file-log collection paths, filter events by service, boot or time, and configure retention for each storage method. Journals are managed by journald, while file logs need to be checked with both program and logrotate operations. This article demonstrates viewing, rotation, compression, and retention results on separate practice journals and files without deleting existing logs.
First distinguish between journal and file logs, and storage space
systemd-journald records service output and system events in structured journals and is viewed using journalctl. rsyslog or applications can record separate text files. Since the same event can be delivered through multiple paths or a program may record in only one location, the absence of a single file does not mean the entire log is missing.
rpm -q systemd logrotate rsyslog
systemctl is-active systemd-journald rsyslog
sudo journalctl --disk-usage
df -hT /var/log
sudo du -xsh /var/log
The source lab environment was Rocky Linux 10.2, systemd 257, logrotate 3.22.0, rsyslog 8.2510.0. The commands were executed on a Linux guest of a dedicated VM. The initially examined base journal usage was 8M, and the directory usage of /var/log was 13M. df shows the entire root filesystem to which that path belongs, so it does not need to match these values.
| How to check | What it shows | Points to distinguish when interpreting |
|---|---|---|
df -hT /var/log |
Capacity and usage of the filesystem to which that path belongs | This is not the size of only the /var/log directory. |
du -xsh /var/log |
Usage of the directory tree within the same filesystem | There may be differences from 'df' depending on permissions, separate mounts, or files that are open but deleted. |
journalctl --disk-usage |
Usage of active and archived files of the selected journal | This is not the combined value including regular text logs and other application data. |
If space is insufficient, first investigate which file system and collection path have grown. Before checking the cause and retention requirements, do not delete the entire /var/log or batch clean the default journal. Export the necessary failure evidence to a safe location, then review the retention policy and actual rotation results.
Prepare a dedicated practice journal separated from the default log.
This time, we will use a journal namespace called lf-course24. A namespace allows you to choose a separate collection instance and storage space, so you can practice without changing the settings of the existing default journal. If you exclude --namespace=lf-course24, the query, rotation, and cleaning targets will differ, so keep it in the subsequent commands.
If a practice-only path already exists, do not overwrite it; check the previous state first. The backup directory and namespace settings below are used only for this practice.
sudo bash <<'BASH'
set -eu
test ! -e /root/lf-course24
test ! -e /etc/systemd/journald@lf-course24.conf.d
mkdir -m 700 /root/lf-course24
cp -a /etc/logrotate.conf /root/lf-course24/logrotate-original.conf
cp -a /etc/logrotate.d /root/lf-course24/logrotate-original.d
systemd-analyze cat-config systemd/journald.conf > /root/lf-course24/journald-original.txt
mkdir -m 755 /etc/systemd/journald@lf-course24.conf.d
BASH
sudoedit /etc/systemd/journald@lf-course24.conf.d/10-course.conf
The full content of the configuration file is as follows. For the supported file paths and option definitions, refer to the official journald.conf manual .
[Journal]
Storage=persistent
SystemMaxUse=16M
SystemKeepFree=16M
SystemMaxFileSize=1M
MaxRetentionSec=1day
ForwardToSyslog=no
Storage=persistent is the setting to select a permanent storage path, and SystemMaxUse · SystemKeepFree are conditions regarding usage and remaining space. SystemMaxFileSize concerns individual file size, and MaxRetentionSec is a setting that limits the retention period. Do not use the small figures from the practice as the recommended values for a production server. They should be determined according to the actual log generation volume, analysis period, and available disk space.
Due to file rotation, deletion units, and active files, momentary usage cannot be interpreted as an upper limit in exact bytes. The source verification checked the settings load and manual rotation/file count management, but did not perform testing for the automatic deletion process after a day or load testing reaching the 16M limit. ForwardToSyslog=no is a setting to disable syslog forwarding for this practice journal, and the default rsyslog settings are maintained.
Open a file with sudoedit /root/lf-course24/write-journal.py and save the following Python example. Python 3 is required, and it outputs 300 log lines differentiated according to arguments 1–4.
import sys
def main():
batch=sys.argv[1]
if batch not in {'1','2','3','4'}:
raise ValueError('Use lab batch 1 to 4')
for number in range(300):
print(f'LF24-BATCH{batch}-{number:03d} '+ 'x'*128, flush=True)
if __name__=='__main__':
main()
The following loop runs four batches as a separate one-time service, rotating after the first three batches. The service’s standard output is sent to the journal of this namespace. Do not use commands to restart the default journal daemon.
bash <<'BASH'
set -euo pipefail
for batch in 1 2 3 4; do
sudo systemd-run --quiet --wait --collect \
--unit="lf-course24-batch${batch}" \
--property=LogNamespace=lf-course24 \
--property=StandardOutput=journal \
/usr/bin/python3 /root/lf-course24/write-journal.py "$batch"
sudo journalctl --namespace=lf-course24 --sync
if test "$batch" -lt 4; then
sudo journalctl --namespace=lf-course24 --rotate
fi
done
BASH
When a new namespace instance starts, it reads dedicated settings. In situations where you are modifying an already running instance, you need to separately consider how the settings are applied and the services currently outputting. Do not blindly repeat this command on an existing lab name.
Find journals by service, boot, time, and priority.
After creating lab logs, try narrowing down the search conditions. -b represents the current boot, -u represents the unit, and --since represents the start time. -n 3 allows viewing the last three lines, and --no-pager lets you skip the pager program.
sudo journalctl --namespace=lf-course24 -b \
-u lf-course24-batch4.service -n 3 --no-pager -o short-iso
sudo journalctl --namespace=lf-course24 -b \
-u lf-course24-batch4.service --since '10 minutes ago' --no-pager -q -o cat
sudo journalctl --namespace=lf-course24 -b \
-u lf-course24-batch4.service -p warning --no-pager -q -o cat
In the source lab, the last logs were LF24-BATCH4-297 ~ 299, and a 10-minute range query immediately after creation produced 300 lines. -p warning selects items of warning priority or higher, and in this test output, there were 0 lines. The absence of search results means there are no items matching the conditions, not that the service is operating normally. Check the namespace, unit name, boot, time range, and permissions together.
When exporting necessary messages to a file, also consider the redirection permissions of the current user's shell. Below is an example of saving to a root-only lab path. Actual operational logs may contain sensitive values such as user information and tokens, so you should check the content and permissions before using them in public posts or for external distribution.
sudo sh -c 'journalctl --namespace=lf-course24 -b -u lf-course24-batch4.service --no-pager -q -o cat > /root/lf-course24/batch4-export.txt'
sudo wc -l /root/lf-course24/batch4-export.txt
In the verification, it was confirmed that the exported message consisted of 300 lines. The definitions of filters and the output, rotation, and cleaning options can be found in the journalctl official manual .
The actual meaning of journal rotation and file count limits
--rotate is the operation of converting the currently active file into an archived file and opening a new active file. Do not confuse rotation itself with the deletion of previous logs. On the other hand, --vacuum-files is a cleanup command that can delete old archived files. The following is executed only on dedicated practice logs that can be deleted.
sudo journalctl --namespace=lf-course24 --disk-usage
sudo journalctl --namespace=lf-course24 --sync
sudo journalctl --namespace=lf-course24 --vacuum-files=1
sudo journalctl --namespace=lf-course24 --disk-usage
In the tested systemd 257 behavior, --vacuum-files=1 did not mean 'leaving one archived file'. The file count limit also affects the active file, but deletion operations apply only to archived files. There are cases where it cannot be reduced to the specified number because the active file cannot be removed.
This practice started with one active file and three archived files containing the previous three batches. When applying a file count limit of 1, all three archived files were deleted, leaving only the active file. Batches 1, 2, and 3 each had 0 lines, and batch 4 in the active file was confirmed to have 300 lines. The usage changed from about 1.5M to 1M.
--disk-usage shows the actual usage of active and archived journals together. Since the logical file size in the file list and the actual allocated space on the disk can differ, the sum of file sizes may differ. Design with capacity, period, and count criteria together, and separately store the necessary logs before deletion.
Verify logrotate with separate files and status files.
Use a separate logrotate configuration for general file logs. Instead of registering in the existing /etc/logrotate.d or sharing system status files, only rotate a dedicated single file. If the following directories already exist, investigate them first without overwriting.
sudo mkdir -m 750 /var/log/lf-course24
sudoedit /root/lf-course24/logrotate.conf
The full content of the configuration file is as follows. size 1k determines at runtime whether the file exceeds the size condition, and rotate 2 specifies keeping two rotated files. Including the current log file, there may be three files, so this should not be confused with the file count option of the journal.
/var/log/lf-course24/application.log {
size 1k
rotate 2
compress
missingok
notifempty
create 0640 root root
}
compress compresses rotated files, and create sets the permissions and ownership of the new current log file. missingok allows missing files, and notifempty prevents rotation of empty files. Since the practice files are written by root, use that ownership, but these settings are not necessarily applied to actual operating service accounts.
Open the file with sudoedit /root/lf-course24/write-file.py and save the entire example for creating the test file below. The current log with existing content is not overwritten, and the next batch is created in the empty file after rotation.
import sys
from pathlib import Path
def main():
batch=sys.argv[1]
if batch not in {'1','2','3'}:
raise ValueError('Use lab batch 1 to 3')
path=Path('/var/log/lf-course24/application.log')
if path.exists() and path.stat().st_size:
raise RuntimeError('Refusing to overwrite a nonempty log')
with path.open('w') as stream:
for number in range(20):
stream.write(f'FILE24-BATCH{batch}-{number:02d} '+ 'y'*128+'\n')
path.chmod(0o640)
if __name__=='__main__':
main()
First, create batch 1 and then check only the execution plan in debug mode.
sudo python3 /root/lf-course24/write-file.py 1
sudo sha256sum /var/log/lf-course24/application.log
sudo logrotate --debug --state /root/lf-course24/logrotate.state /root/lf-course24/logrotate.conf
sudo sha256sum /var/log/lf-course24/application.log
In this verification, the front and rear hashes were the same, and the compressed and state files were not created. Even if the debug output shows plans like renaming or Creating new state, it does not mean the actual changes have been completed. The actual execution and state file options are distinguished in logrotate official manual .
Now, actually rotate batch 1, then create and rotate batches 2 and 3 in order. Since this is a practice based on size, the forced execution option is not used.
sudo logrotate --verbose --state /root/lf-course24/logrotate.state /root/lf-course24/logrotate.conf
for batch in 2 3; do
sudo python3 /root/lf-course24/write-file.py "$batch"
sudo logrotate --verbose --state /root/lf-course24/logrotate.state /root/lf-course24/logrotate.conf
done
sudo stat -c '%a %U:%G %s %n' /var/log/lf-course24/application.log /var/log/lf-course24/application.log.1.gz /var/log/lf-course24/application.log.2.gz
sudo gzip -cd /var/log/lf-course24/application.log.1.gz | head -n 1
sudo gzip -cd /var/log/lf-course24/application.log.2.gz | head -n 1
The actual result was that the current file was 0 bytes, permission 640, owner root, with 20 lines of batch 3 remaining in .1.gz and 20 lines of batch 2 remaining in .2.gz. The oldest batch 1 was removed. When executed again on the empty current file, it was not rotated.
This setting is the condition that logrotate checks when it runs. Specifying size 1k does not always keep the file under 1 KiB. In actual operation, you need to check both the execution interval of the schedule and the amount generated. This exercise is a manual execution verification and has not changed the system timer or global settings.
Apply rotation to running services and clean up the lab
Since the one-time test program finishes writing and exits, even if you rename the file and create a new one, the next batch will use the new file. A continuously running daemon may continue to write to the previously opened file. You need to check the official reopen/reload method of the operational service and features such as postrotate in logrotate, and you should not extend this test result to verify daemon log reopening.
copytruncate is not a solution suitable for every program. Review the conditions in the manual, such as the possibility of loss between copy and cut operations. Instead of indiscriminately deleting existing logs or forcibly stopping the service to avoid problems, check the service's logging method along with file permissions, SELinux labels, and post-rotation behavior.
When finishing the exercise, stop only the dedicated journal instance and socket. There is no need to stop the default systemd-journald.service or rsyslog.
sudo journalctl --namespace=lf-course24 --sync
sudo systemctl stop systemd-journald@lf-course24.service systemd-journald@lf-course24.socket systemd-journald-varlink@lf-course24.socket
sudo cp -a /etc/systemd/journald@lf-course24.conf.d/10-course.conf /root/lf-course24/journald-namespace.conf
sudo rm -- /etc/systemd/journald@lf-course24.conf.d/10-course.conf
sudo rmdir /etc/systemd/journald@lf-course24.conf.d
systemctl is-active systemd-journald rsyslog
Test logs, exports, and status files are kept as practice records if needed. In the source verification, the log paths and file log lists in the dedicated namespace were checked, and after comparing whether the backup hashes were the same, only the corresponding dedicated originals were organized. The default journal settings, existing logrotate settings, individual configuration files, and fstab matched those before the work, and the default journald, rsyslog, SSH, and firewall were active.




