Linux Tutorial / Investigate Linux Failures with Logs
To investigate a failure, identify the affected service and time, collect related entries from the same boot and timezone, and compare them with a reproduction. An ERROR line or the last message alone is not enough to establish the cause.
Review basic service state in the systemd introduction. This lesson correlates times, states, and logs; apply the method in a controlled environment with failure and recovery practice.
System configuration exercises use a conventional Ubuntu 26.04 LTS or Rocky Linux 9 VM as the reference environment. Containers, WSL, and other releases can differ in service and package configuration. Prepare console recovery and record your versions using the virtual machine setup lesson.
Record the symptom, time, and recent changes
Replace “the server is broken” with a precise observation such as “HTTPS requests from client A have returned 502 since 14:03.” Compare client and server clocks and timezones, and record the last known good state and recent deployments, configuration changes, or reboots. Keep observed facts separate from hypotheses.
date --iso-8601=seconds
timedatectl
systemctl --failed
journalctl --list-bootsIf a previous boot is absent, its journal may not have been retained or may have been removed. Inspect volatile versus persistent storage and retention limits. Missing entries do not establish that no failure occurred, and limited permissions can hide relevant records.
Filter the journal by service, boot, and time
Replace NAME.service with the actual unit. Inspect status, read its current-boot journal, then narrow the time window. A focused saved interval is easier to compare than an unbounded stream of unrelated messages.
systemctl status NAME.service --no-pager
journalctl -b -u NAME.service --since "30 minutes ago" --no-pager
journalctl -b -p warning --since "30 minutes ago" --no-pager
journalctl -k -b --since "30 minutes ago" --no-pager-u selects a unit, -b a boot, and -k kernel messages. -p warning includes warning and more severe priorities, but an application may label entries inconsistently or write clues as ordinary stdout. Compare the unfiltered interval after using priority filters.
Find a deliberately created logger event
The following writes two tagged messages without changing a service. It requires a Linux environment where logger output is collected into the journal. Restricted containers or systems without a syslog socket may behave differently; check the collection path before interpreting an empty result.
tag="linux-course-log-$(id -u)"
logger -t "$tag" "stage=start case=practice"
logger -p user.warning -t "$tag" "stage=check result=sample-warning"
journalctl -t "$tag" --since "5 minutes ago" --no-pagerLocate both messages and their timestamps, then add -p warning and observe whether the ordinary-priority message is excluded. The sample warning is deliberately generated, not a real incident. Use authorized journal-reading privileges if needed; do not erase production logs or restart collection merely to make the exercise work.
Correlate messages with independent evidence
| Clue | Independent checks |
|---|---|
| Permission denied | Ownership, path permissions, SELinux labels |
| No space left on device | df -h and df -i on the affected mount |
| Address already in use | Socket ownership and duplicate processes |
| Connection refused or timeout | Listener, route, filtering, dependency |
| Repeated restart or failure | Unit configuration, exit status, recent deployment |
A disk-full message can concern the application’s target path rather than the disk storing the log. Permission denial can arise from parent-directory traversal or policy, not just the file mode. Inspect the actual referenced resource rather than masking the problem with root execution or world-writable permissions.
Inspect file logs and retention
Applications may log to files under /var/log or to a central collector rather than the journal. Confirm the configured path before using tail or grep. Include rotated and compressed records as needed and normalize timestamp formats and timezones. journalctl --disk-usage reports journal storage usage.
Deleting or vacuuming logs during an investigation can destroy evidence. Preserve the needed interval in access-controlled storage and redact tokens, cookies, personal data, and private addresses before sharing it. Record the cause, targeted change, repeated test, and remaining uncertainty, then retain evidence according to policy.
