Story / scenario opener
You applied a system update on Fedora Silverblue and the next boot drops you to a black screen or a login loop. The desktop environment refuses to start, or a kernel panic flashes before the system hangs. You need to get back to a working state immediately without wiping your data or reinstalling.
What is actually happening
Silverblue does not patch files in place. Traditional distributions modify /usr/bin and /etc directly, which means a bad package can leave your system in a half-broken state. Silverblue uses an immutable bootc model. Every update downloads a complete, verified root filesystem image and mounts it as a new deployment. Your running system stays exactly as it was until you reboot. When an update breaks something, the damage is isolated to that new deployment. The old deployment still sits on disk, untouched and ready to boot.
The underlying mechanism relies on ostree commits. Each Fedora release and every update is a cryptographic snapshot of the entire filesystem. The bootloader points to the commit hash you want to run. Your home directory and configuration files live on a separate overlay layer that mounts on top of whichever base image you select. This separation is why you can switch between deployments instantly. The base image changes, but your data stays mounted and accessible.
The GRUB menu lists every deployed image. You just need to tell the bootloader to use the older one. Reboot before you debug. Half the time the symptom is gone.
The fix: rolling back from GRUB
Restart the machine. Hold Shift during the early boot phase to interrupt the automatic countdown. The GRUB menu appears with a list of Fedora Silverblue entries. Each entry shows a commit hash and a timestamp. The top entry is the broken update. The second entry is your previous working state. Highlight the previous deployment and press Enter. The system boots into the older image. Your user data, home directory, and configuration files remain intact. The broken update is still on disk, but it is no longer active.
Some UEFI firmware hides the GRUB menu behind a fast boot splash screen. If you miss the window, hold Shift longer or press Esc repeatedly during the BIOS handoff. You can also adjust the GRUB timeout in /etc/default/grub if you want the menu to appear by default. Edit the GRUB_TIMEOUT value and run sudo grub2-mkconfig -o /boot/grub2/grub.cfg to apply the change. Note that on immutable systems, /etc is writable, but /usr/lib contains package defaults. Always edit /etc. Never edit /usr/lib.
Booting the old deployment from GRUB is temporary. The next automatic update or reboot might switch you back to the broken image. You need to tell the system to make the older deployment the default. Run journalctl first. Read the actual error before guessing.
Making the rollback permanent with rpm-ostree
Open a terminal and check the deployment stack. The rpm-ostree tool manages the transactional update pipeline and tracks which images are available on disk.
rpm-ostree status
# WHY: Shows the current booted deployment and the full list of available images on disk.
# WHY: The output marks the active deployment with a star.
# WHY: Each entry includes a commit hash, deployment index, and timestamp.
Identify the commit hash of the working version. Run the rollback command to swap the default boot target.
sudo rpm-ostree rollback
# WHY: Swaps the default boot entry to the previous deployment.
# WHY: Does not delete the broken image. It just changes the bootloader priority.
# WHY: Requires a reboot to take effect.
If the immediate previous deployment is also broken, or if you need to jump back three versions, use the deploy command with a specific hash.
sudo rpm-ostree deploy <commit-hash>
# WHY: Re-downloads or re-activates a specific historical image.
# WHY: Places the target image at the top of the boot menu.
# WHY: Safe to run multiple times without corrupting the disk layout.
Reboot the system. The bootloader will now default to the restored image. The transactional model ensures that partial updates never leave the disk in an inconsistent state. Trust the package manager. Manual file edits drift, snapshots stay.
Diagnosing the broken deployment
You might need to understand why the update failed before you can safely upgrade again. Silverblue supports rescue mode for any deployment. Restart and interrupt GRUB again. Highlight the broken deployment and press e to edit the kernel parameters. Find the line starting with linux. Move the cursor to the end of that line and append systemd.unit=rescue.target. Press Ctrl+X to boot. The system drops you into a root shell with minimal services running.
journalctl -xb -p 3
# WHY: Filters the current boot log to show only critical and error level messages.
# WHY: Skips the noise of successful service startups.
# WHY: Helps you isolate the exact unit or module that crashed.
Most sysadmins type journalctl -xeu <unit> muscle-memory style to jump straight to the failing service. The x flag adds explanatory text and the e flag jumps to the end. Combine it with -b to scope the output to the current boot.
Check for overlay or layer conflicts. If you installed third-party packages with rpm-ostree install, they might conflict with the new base image. List your active layers.
rpm-ostree package list
# WHY: Shows every manually installed package overlaying the base image.
# WHY: Conflicts here usually cause the boot failure or service crash.
# WHY: Output is sorted by repository and package name.
Remove the problematic layer if you found a culprit.
sudo rpm-ostree uninstall <package-name>
# WHY: Strips the conflicting overlay from the current deployment.
# WHY: Generates a new boot entry with the layer removed.
# WHY: Requires a reboot to apply the clean image.
SELinux denials often surface during broken boots. Check journalctl -t setroubleshoot for one-line summaries before disabling the security policy. Read those before disabling SELinux.
Verify it worked
Confirm the system is running the correct image and that the boot target is locked in.
rpm-ostree status
# WHY: Verifies the star marker is on the restored deployment.
# WHY: Confirms the broken image is still listed but marked as inactive.
# WHY: Shows the exact commit hash currently in use.
Check that your desktop environment and network services are responding normally. If the system boots cleanly and your workflow is restored, you can safely ignore the broken deployment until the maintainers release a fix. Snapshot the system before the upgrade. Future-you will thank you.
Common pitfalls and what the error looks like
The GRUB menu sometimes hides behind a fast boot splash screen. If you miss the window, hold Shift longer or press Esc repeatedly during the BIOS handoff. Some UEFI setups require you to disable Fast Boot in the firmware settings to see the menu reliably.
You might see Failed to start GNOME Display Manager or systemd[1]: Starting Network Manager... followed by a hang. These are service-level failures inside the new deployment. They do not mean your disk is corrupted. They mean the new image contains a broken configuration or a conflicting package layer. Rescue mode isolates the problem without touching your data.
Another common issue is garbage collection removing the old deployment. rpm-ostree automatically cleans up old images to save disk space. If you wait too long before rolling back, the previous commit might be gone. Pin important deployments before they age out.
When to use this vs alternatives
Use GRUB rollback when you need immediate recovery and cannot access the desktop or terminal. Use rpm-ostree rollback when you want to permanently switch the default boot target to the previous working state. Use rpm-ostree deploy <hash> when you need to jump back multiple versions or restore a specific historical snapshot. Use rescue mode when you must diagnose logs or remove a conflicting package layer. Use rpm-ostree uninstall when a manually added package is breaking the boot sequence. Stay on the upstream Workstation if you only deviate from the defaults occasionally.