Rocky Linux 10 Tutorial / Diagnose boot, initramfs and network failures
When Rocky Linux 10 does not boot or cannot be accessed via SSH, you must first identify the stage where it stopped. The materials to verify and methods to recover differ between bootloader, kernel, initramfs issues, and post-login network problems. This tutorial helps you learn the diagnostic sequence and practice error detection and recovery using a separate initramfs file and isolated network.
Identify the last successful boot stage
Do not assume the operating system has not booted just because SSH connection fails. Check the actual screen on the VM console or remote management console, and record the last displayed message and the point of failure. Also check whether it was immediately after an update, following changes to the disk or virtual hardware, or after network configuration changes.
| Observed state | First objects to check | Initial response |
|---|---|---|
| Cannot find boot device or GRUB menu | Firmware boot order, disk connections, EFI boot entries | Check virtual hardware/device status without initializing the disk. |
| Hangs or panics immediately after kernel messages | Selected kernel, virtual hardware, drivers, and error messages | Preserve errors on the console and check if it can boot once with a previously installed kernel. |
| Cannot find root device in dracut shell | Root device, UUID, LVM, drivers, and initramfs | Compare device recognition and boot parameters and examine the initial image of the corresponding kernel. |
| systemd emergency mode / mount failure | fstab, missing disks, failed mount units | Check the failed unit and the actual UUID, and restore the settings before the change. |
| Console login works, but SSH connection fails. | Address, routes, DNS, sshd and firewall | Check the IP and listening port from the console and inspect layer by layer. |
GRUB is a bootloader that selects and executes the kernel to boot. initramfs is an initial memory filesystem that contains the drivers and tools needed before using the actual root filesystem. For example, if the root is on LVM, the process of finding the device and activating the logical volume is required first. The initial boot description of the dracut project explains this role and the difference between host-only images and generic images.
Before performing recovery work, secure a valid backup, maintenance window, and a means of console access. Avoid executing arbitrary recovery or format commands on a mounted root filesystem or turning off security features first just to reduce boot issues. Save readable logs and current settings first to be able to compare before and after changes.
Diagnostic data to save when login is possible
The command below checks the current kernel, the default boot entry, the actual root mount, and failed units. uname -r is the currently running kernel, and grubby --default-kernel is the kernel set as the default boot target. The two values being different does not immediately indicate an error. You may have manually selected a previous kernel or not yet rebooted after installing a new kernel.
uname -r
cat /proc/cmdline
rpm -q kernel-core dracut grubby
sudo grubby --default-kernel
sudo grubby --info=ALL
findmnt /
sudo findmnt --verify --verbose
systemctl --failed --no-pager
sudo journalctl -b -p warning --no-pager
sudo journalctl --list-boots
systemd-analyze time
If previous boot records are listed, view warnings for that boot with sudo journalctl -b -1 -p warning --no-pager. If there is no persistent journal or if the retention period has passed, there may be no previous records. Even if systemctl --failed is empty, it does not mean there are no warnings in the kernel.
The source VM was running with 6.12.0-211.74.1.el10_2.x86_64, and the default kernel was the same. The previous kernel 6.12.0-211.16.1.el10_2.0.1.x86_64 and rescue entries were also present. This path and version are example checks and are not values to be entered directly in your own environment.
Additionally, the previous boot recorded about 4 minutes 57 seconds at the kernel stage, and about 5 minutes 22 seconds in total. At that time, there were warnings for AES-NI cryptographic operation self-check timeouts and kernel alerts. Do not consider those warnings normal just because login later succeeded, or immediately attribute the same symptoms to initramfs corruption. Boot time, console, and kernel logs are viewed together to narrow down the delay location.
Temporarily boot with the previous kernel and narrow down the cause.
If the problem appears only on the new kernel after an update, you can select the previously installed kernel to compare. Preserving the existing kernel and verifying the update connects to the processes covered in Kernel Update, Reboot, and Recovery Guide . If a previous kernel is not installed, it is assumed not to appear in the GRUB menu.
- Record the current console screen and restart during maintenance time. If the bootloader menu is hidden, use the method to display the menu in the VM or firmware environment.
- In the GRUB menu, select the exact kernel version entry before the error occurred. Distinguish between the rescue entry and regular previous kernel entries.
- After logging in, verify with
uname -rthat it is indeed a previous version, and recheck disks, network, and services. Do not assume that successful booting alone means that all business functions are normal. - After confirming that the issue reproduces only on the new kernel, investigate the package status, related drivers, and known issues. Do not delete the working kernel first.
Selecting to boot once and changing the default kernel are different tasks. If you need to change the default, set it to the exact kernel path confirmed in grubby --info=ALL, and record the original default path so it can be reverted. The distinction between checking the default kernel and verifying the running version after reboot is also explained in Rocky Linux official kernel practice . The actual installed version and file path need to be checked separately on this server.
In the dracut shell, boot parameters and devices are compared with cat /proc/cmdline and lsblk -f. If LVM tools are available, check the root volume with lvs. If a /run/initramfs/rdsosreport.txt has been generated, preserve the associated error report as well. If the disk itself is not visible, check the controller and virtual disk connection before recreating the image.
Detecting corruption of a separate initramfs file and regenerating it.
The exercises in this section are performed on a dedicated Rocky Linux 10 VM that boots normally. Without changing the actual /boot image, only separate files are created. The source exercise used dracut 107-9.el10_2, and the failure of analyzing the incorrect file and the module list of the regenerated image were confirmed through the following procedure.
sudo -i
set -eu
umask 077
test ! -e /var/tmp/lf-course30
install -d -m 700 /var/tmp/lf-course30
k=$(uname -r)
test -d "/usr/lib/modules/$k"
sha256sum "/boot/initramfs-$k.img" > /var/tmp/lf-course30/boot-before.sha256
grubby --default-kernel > /var/tmp/lf-course30/default-before.txt
cp --preserve=all "/boot/initramfs-$k.img" /var/tmp/lf-course30/original.img
lsinitrd -m /var/tmp/lf-course30/original.img
printf 'deliberately invalid LF30 initramfs copy\n' > /var/tmp/lf-course30/broken.img
if lsinitrd -m /var/tmp/lf-course30/broken.img; then
echo 'Unexpected parse success: stop and inspect this test'
exit 1
else
echo 'Expected rejection of the invalid test file'
fi
dracut --force "/var/tmp/lf-course30/rebuilt-$k.img" "$k"
lsinitrd -m "/var/tmp/lf-course30/rebuilt-$k.img"
sha256sum -c /var/tmp/lf-course30/boot-before.sha256
grubby --default-kernel > /var/tmp/lf-course30/default-after.txt
cmp /var/tmp/lf-course30/default-before.txt /var/tmp/lf-course30/default-after.txt
The incorrect test file failed to be analyzed along with File format not recognized. The regenerated file was about 39 MiB, and the module list such as lvm, rootfs-block, kernel-modules could be read. The last original checksum was OK, and the base kernel was the same. Simply being able to read the file size or module list does not prove that the image will actually boot successfully. In this exercise, the new image was not linked to a boot entry nor was the system rebooted with that file.
In dracut --force, the first path is the output file and the next value is the target kernel version. Consult the dracut command manual and lsinitrd manual. Since the installed distribution version may differ from the version in the project's latest documentation, check the currently supported options with dracut --help and lsinitrd --help.
Conditions for applying to actual boot image recovery
If you need to actually restore the initramfs, first secure a recovery shell using a previously functioning kernel or a trusted installation media of the same series. When entering the troubleshooting or Rescue menu of the installation media, check the mount location and read/write status of the existing system as indicated on the screen. If the path is shown as /mnt/sysroot, then verify etc/os-release and the mount information under it, and use the tools of the system installed at chroot /mnt/sysroot. You should first confirm that the root, separate /boot, and EFI partitions are connected to the correct devices.
An important point is that uname -r of the recovery shell may be the running kernel of the recovery environment rather than the installed target kernel. Compare the installed /usr/lib/modules, /boot/vmlinuz-*, and boot entries to select the version to restore. If an image is created in an ordinary directory with the same name while /boot is not mounted, the actual boot partition will not be fixed.
bash <<'RECOVERY'
set -euo pipefail
findmnt /boot
df -h / /boot
df -i / /boot
ls /usr/lib/modules
grubby --info=ALL
read -r -p 'Exact installed target kernel version: ' target_k < /dev/tty
test -n "$target_k"
test -d "/usr/lib/modules/$target_k"
test -f "/boot/vmlinuz-$target_k"
test -f "/boot/initramfs-$target_k.img"
backup="/boot/initramfs-$target_k.img.before-$(date +%Y%m%d-%H%M%S)"
test ! -e "$backup"
cp --preserve=all "/boot/initramfs-$target_k.img" "$backup"
dracut --force "/boot/initramfs-$target_k.img" "$target_k"
lsinitrd -m "/boot/initramfs-$target_k.img"
RECOVERY
This is a recovery procedure that replaces a real boot file . Apply it only after diagnosing the cause and securing a backup and console. It differs from the separate-file lab above; the source validation did not execute the replacement command. test -f checks the existing image in this regeneration scenario. If the image is entirely missing, first verify the available good copy and target kernel, then choose a procedure appropriate to that missing-image condition.
The Rocky Linux initramfs regeneration guide describes preserving the existing image before rebuilding it. The block above assumes the expected separate /boot is mounted and runs in the installed system's root shell. If the recorded layout differs, adapt the inspection before considering a write. After checking the results, exit the recovery shell and restart the system to boot from the disk, then verify the running kernel, root mount, and key services. If it fails, try accessing the previously working kernel and restore the preserved files and settings. Do not mix the kernel version and image version from the boot entry when copying.
If the cause of emergency mode is an additional missing disk or an incorrect fstab, regenerating initramfs may not resolve it. The process of restoring the original settings by comparing findmnt --verify, lsblk -f with the failed mount units is described in UUID·fstab automatic mount and boot failure prevention guide . Choose the filesystem check and repair tools after confirming the type of filesystem and mount status.
Network diagnosis when login is successful but connection is not
After console login, separate the device, address, routes, name resolution, and service reception status. Having an address is different from having a communication path, and connecting via IP is a different check from connecting via hostname.
ip -br link
ip -br address
ip route
nmcli device status
nmcli -t -f NAME,UUID,DEVICE connection show --active
nmcli device show
getent ahostsv4 www.rockylinux.org
sudo systemctl status sshd --no-pager
sudo ss -lntp
sudo firewall-cmd --get-active-zones
sudo firewall-cmd --list-all
If the interface is DOWN, check the cable, virtual NIC connection, and the corresponding profile. If there is no IPv4 address, set DHCP or a static address; if there is no external route, check the default gateway. ip route get 192.0.2.10 shows the route the kernel would choose without sending packets. Replace the documentation address with the destination you are diagnosing. If the current management interface is brought down or the profile is disabled on a remote SSH server, that connection will also be lost, so make configuration changes from the console or after securing a verified automatic recovery method.
If you can connect via IP and port but name resolution fails, start by narrowing it down to DNS. Compare the getent results of the server with NetworkManager's DNS settings and name resolution path. A single ping failure should not be taken to mean that all IP communication has failed. Even if ICMP is blocked, necessary TCP services can still communicate. The safe address and DNS change and restoration process can be continued and checked in NetworkManager checkpoints and the static address and DNS tutorial .
Even if sshd is normal, connections may fail if accessed with the wrong address or if the receiving address is restricted. Match the local receiving address/port in ss with the actual connection target, and check the firewall based on the zone to which the device belongs. Instead of taking down the entire firewall when the cause is unclear, only compare the necessary paths and services.
Network failure and recovery exercises that do not interfere with management connections.
A network namespace is an isolated space that has its own interfaces, addresses, routes, and so on. Here, we create client and server spaces inside a VM and connect them directly with a virtual Ethernet pair (veth). Management enp0s3 or the actual NetworkManager profile is not changed. Python HTTP services serving as the server run only under the lfstudent account and do not use external Internet or DNS.
Execute in the administrator shell opened in the previous section. ip, python3, curl, and lfstudent accounts are required. If there are paths, namespaces, or units with the same name already existing, do not reuse or delete them; stop the process instead.
test ! -e /srv/lf-course30-network
test ! -e /run/netns/lf30-client
test ! -e /run/netns/lf30-server
install -d -m 755 /srv/lf-course30-network
printf 'LF30 network restored\n' > /srv/lf-course30-network/index.html
chmod 644 /srv/lf-course30-network/index.html
ip netns add lf30-client
ip netns add lf30-server
ip link add lf30c netns lf30-client type veth peer name lf30s netns lf30-server
ip -n lf30-client address add 198.18.30.1/30 dev lf30c
ip -n lf30-server address add 198.18.30.2/30 dev lf30s
ip -n lf30-client link set lo up
ip -n lf30-server link set lo up
ip -n lf30-client link set lf30c up
ip -n lf30-server link set lf30s up
systemd-run --unit=lf-course30-network --uid=lfstudent \
--property=Type=exec \
--property=NetworkNamespacePath=/run/netns/lf30-server \
/usr/bin/python3 -m http.server 18030 --bind 198.18.30.2 \
--directory /srv/lf-course30-network
sleep 1
ip netns exec lf30-client curl --fail --silent --max-time 3 \
http://198.18.30.2:18030/
ip -n lf30-client route get 198.18.30.2
Normal response is LF30 network restored, and dev lf30c src 198.18.30.1 appears in the path. The following is the process of bringing down only the practical interface of the client.
ip -n lf30-client link set lf30c down
ip -n lf30-client -br address
ip -n lf30-client route
if ip netns exec lf30-client curl --fail --silent --show-error \
--max-time 3 http://198.18.30.2:18030/; then
echo 'Unexpected success: inspect the test'
else
echo "Expected connection failure: $?"
fi
ip -n lf30-client link set lf30c up
ip -n lf30-client route get 198.18.30.2
ip netns exec lf30-client curl --fail --silent --max-time 3 \
http://198.18.30.2:18030/
In the source practice, the interface state was DOWN even though the address remained, and the route list was empty. curl failed to connect with exit status 7. After bringing it back up, the direct connection paths returned and read the same content. This compares the cause and recovery results by changing only one link status while keeping the same service and address. This result does not reproduce examinations of DNS, public routing, or external firewall failures.
Cleanup and determination of recovery completion
Before deleting the practice namespace, first stop any services running within it. The deletion targets below are only the two names newly created in this lecture. Once the namespace is removed, the virtual links within it are also cleaned up.
systemctl stop lf-course30-network.service
ip netns delete lf30-client
ip netns delete lf30-server
test ! -e /run/netns/lf30-client
test ! -e /run/netns/lf30-server
ip -br address
systemctl is-active sshd firewalld
getenforce
exit
In the source VM, the existing boot images, kernel files, boot entries, GRUB environment files, as well as fstab, hosts, administrative NetworkManager profiles, and firewall settings hashes remained unchanged. The default kernel was the same, and the management interface, SSH, and firewall were preserved. Separate copies of initramfs and practice web materials were kept along with verification records.
The completion criteria for operational recovery are not that the console screen has changed, but that the system boots with the intended kernel, required disks are properly mounted, and administrative access and work services are functioning normally. Recording applied changes, inspections confirmed as normal, remaining warnings, and rollback methods allows you to continue the basis for judgment if the same failure occurs again.




