The scenario
You set up a Podman Machine to run containers locally. It works fine for a few weeks. Then you notice the base image is still Fedora Workstation, which pulls in desktop packages you never use. You want a leaner, immutable base like Universal Blue, or you built a custom atomic image with your own pre-installed tools. You need to swap the underlying operating system without wiping your container storage or breaking your network setup.
What os apply actually does
Podman Machine runs a lightweight virtual machine to host your containers. By default, it provisions a standard Fedora Workstation image. The os apply command replaces that guest operating system with a different container image formatted as an OS. Think of it like swapping the foundation of a house while keeping the furniture inside. Your containers, volumes, and network bridges stay mounted. The command pulls the new image, writes it to the VM disk, and reboots the machine into the new base.
This works because modern immutable distributions package the entire OS as a single container image. Universal Blue ships exactly in that format. Custom atomic images follow the same layout. The Podman Machine tool knows how to mount the ostree deployment layer and hand control to the new bootloader. The command reads the image manifest, extracts the root filesystem, and writes it to the QEMU disk partition. It then updates the bootloader configuration to point at the new deployment. The container runtime socket remains at the same path. Your existing containers resume without reconfiguration.
Convention aside: Podman Machine stores its VM disk in ~/.local/share/containers/podman/machine/qemu/. The os apply command modifies that disk in place. Back up the directory before you run it. A corrupted disk image means rebuilding the machine from scratch.
How to rebase your Podman Machine
First, check which machine is currently active. Podman defaults to podman-machine-default, but you might have named yours differently.
podman machine list
# WHY: Shows active state, VM name, and current OS image reference
# WHY: Confirms the target machine is running before you attempt a rebase
# WHY: Reveals disk usage so you can verify free space for the new image
If the machine is stopped, start it. The os apply command requires a running guest to mount the new image correctly.
podman machine start
# WHY: Boots the QEMU VM so the disk is accessible for modification
# WHY: Skips this step if the machine is already in a Running state
# WHY: Waits for the systemd init sequence to complete before returning
Stop any long-running containers before the swap. The reboot will interrupt active processes.
podman stop --all
# WHY: Gracefully shuts down every running container on the host
# WHY: Prevents data corruption in volumes that are actively writing
# WHY: Ensures the container runtime socket is idle during the reboot
Now run the rebase command. Replace the image reference with your target. Universal Blue uses a public registry. Custom images might live on localhost or a private registry.
podman machine os apply quay.io/universal-blue/machine-os:latest --restart
# WHY: Pulls the Universal Blue base image and writes it to the VM disk
# WHY: --restart forces an immediate reboot into the new OS layer
# WHY: Omit --restart if you want to inspect the disk before rebooting
If you are using a locally built image, point to the containers storage transport.
podman machine os apply containers-storage:localhost/my-custom-os:latest --restart
# WHY: Reads the image directly from the local container store
# WHY: Bypasses network pulls and registry authentication
# WHY: Requires the image to already exist in podman's local cache
The command will download the image, unpack the filesystem layers, and reboot the VM. Wait for the prompt to return. Do not interrupt the process. The disk rewrite takes thirty to sixty seconds depending on your connection speed. The tool will print progress indicators for each layer. If the network drops mid-pull, the command aborts and leaves the disk in a safe state.
Snapshot the machine disk before the rebase. Future-you will thank you when the network bridge fails to come up.
Verify the new base is running
After the reboot, confirm the guest is actually running the new base. SSH into the machine and check the OS release file.
podman machine ssh cat /etc/os-release
# WHY: Connects to the VM and prints the distribution identifier
# WHY: Confirms the bootloader handed control to the new ostree deployment
# WHY: Shows PRETTY_NAME and ID fields that match your target image
You should see ID=universal-blue or your custom distribution name. Check the kernel version to ensure the new base is using its own kernel packages.
podman machine ssh uname -r
# WHY: Prints the running kernel version string
# WHY: Validates that the new OS image provided its own kernel
# WHY: Mismatched kernels indicate the old boot layer is still active
Verify that the container runtime socket is listening. The new base must include the podman.socket systemd unit.
podman machine ssh systemctl is-active podman.socket
# WHY: Checks the systemd state of the container runtime socket
# WHY: Returns active if the service started successfully on boot
# WHY: Fails silently if the unit is missing from the new base image
Run a quick container test to verify the container runtime still talks to the host.
podman run --rm alpine echo "Container runtime is functional"
# WHY: Spins up a temporary container to test the CRI socket
# WHY: Confirms the new base image includes the required systemd units
# WHY: Fails fast if the container service failed to start on boot
Run the container test before you migrate production workloads. A missing podman.socket unit will break everything.
Common pitfalls and error patterns
The rebase process fails in three predictable ways. Network configuration drift is the most common. The new base image might expect a different interface name or DHCP client. If the VM loses connectivity, the container network bridge cannot route traffic.
Check the journal inside the machine for network manager failures.
podman machine ssh journalctl -xeu NetworkManager
# WHY: Filters logs for the network manager service
# WHY: The -x flag adds explanatory hints to systemd error codes
# WHY: The -e flag jumps to the end of the log buffer
If you see [FAILED] Failed to start NetworkManager.service, your interface naming scheme changed. Immutable images often use predictable network interface names. Edit the network configuration in /etc/NetworkManager/system-connections/ inside the VM. Never edit files in /usr/lib/. Those paths ship with the package and get overwritten on the next layer update.
Storage permission errors appear when the new base image runs a different SELinux policy. Container volumes mounted from the host might get denied access. Check the audit log for denials.
podman machine ssh ausearch -m AVC -ts recent
# WHY: Queries the audit subsystem for recent access vector cache denials
# WHY: Shows exactly which process was blocked and which file triggered it
# WHY: Use this before disabling SELinux entirely
SELinux denials show up in journalctl -t setroubleshoot with a one-line summary. Read those before disabling the policy. Most volume permission issues resolve by running restorecon -Rv /var/lib/containers on the host. The command resets the security contexts to match the new base image expectations.
The third failure mode is an incomplete image pull. If your network drops during the os apply download, the disk gets a partial write. The VM will hang at the bootloader. Rescue it by booting from a live ISO or recreating the machine. You can also force a rollback by running podman machine os rollback if the tool kept a previous deployment layer.
Run journalctl -xe first. Read the actual error before guessing.
When to rebase vs when to stick with defaults
Use Universal Blue when you want a community-maintained immutable base that receives faster security patches than standard Fedora. Use a custom atomic image when you need pre-baked development tools, specific kernel parameters, or hardened security profiles baked into the base layer. Use the default Fedora Workstation image when you need maximum compatibility with desktop applications and proprietary drivers. Stay on the upstream Workstation if you only deviate from the defaults occasionally.
Trust the package manager. Manual file edits drift, snapshots stay.