lib: Add experimental unified storage support for install #1601

jmarrero · 2025-09-09T17:28:09Z

lib: Add experimental unified storage support for install

Add an experimental --experimental-unified-storage flag to bootc install
that uses bootc's container storage (/usr/lib/bootc/storage) to pull
images first, then imports from there. This is the same approach used
for logically bound images (LBIs).

Background:
The unified storage approach allows bootc to share container images with
podman's storage, reducing disk space and enabling further integration
with the Podman ecosystem.

Changes:

Add --experimental-unified-storage CLI flag to install subcommands
Add sysroot_path parameter to prepare_for_pull_unified() and pull_unified()
to handle the different mount points during install vs upgrade/switch
Handle localhost images specially by exporting from ostree to container
storage first (these can't be pulled from a registry)
Skip pull in prepare_for_pull_unified() if image already exists in
bootc storage
Add TMT test for install with unified storage flag
Add TMT test for switching to unified storage on running system

The sysroot_path fix is needed because during install the target disk
is mounted at a specific path (e.g., /var/mnt), not /sysroot. Skopeo
needs the actual filesystem path to find the bootc storage.

Relates: #20

Assisted-by: Claude Code (Sonnet 4.5 & Opus 4.5)

[core@cosa-devsh ~]$ sudo podman --storage-opt=additionalimagestore=/usr/lib/bootc/storage images
REPOSITORY                       TAG         IMAGE ID      CREATED       SIZE
quay.io/jmarrero_rh/soft-reboot  1           97f84fcb062e  25 hours ago  1.83 GB

cgwalters

Thanks for working on this!

cgwalters · 2025-09-09T18:20:03Z

crates/lib/src/cli.rs

+    /// Use podman/skopeo to pull image to additionalimagestore, then read from container storage.
+    /// This provides a unified approach that leverages existing container tooling.
+    #[clap(long)]
+    pub(crate) unified: bool,


I don't think this should be a CLI flag that operates just once; it should be something like bootc image --set-unified or so and act persistently.

Also we should support setting this at install time so that it happens from the very start.

(hmm my bad I thought we discussed this but I failed to update the issue #20 or something maybe?)

Ahh man yeah, we talked about saving the config on the origin file IIRC, I forgot.

cgwalters · 2025-09-09T18:20:41Z

crates/lib/src/podman.rs

+    use bootc_utils::CommandRunExt;
+
+    // Use podman pull with additionalimagestore pointing to bootc storage
+    let bootc_storage_path = "/usr/lib/bootc/storage";


Use podstorage.rs instead please

To elaborate on this, an important aspect here is currently the GC of the bootc/storage instance is rooted in the set of LBIs. That will need to be extended to include the host image.

Another big task related to this - we'll eventually really want to use the podman HTTP APIs so we can properly monitor progress. Right now we are very crudely relying on doing this synchronously to the existing tty and reusing the podman CLI tty output. But that can't really work if we ever need to render our own progress APIs.

https://lib.rs/crates/bollard might be a good choice for this?

crates/lib/src/deploy.rs

crates/lib/src/cli.rs

jlebon · 2025-10-07T13:31:09Z

Do we get reflinks when importing from the alternate storage into the ostree repo?

cgwalters · 2025-10-07T14:36:03Z

Do we get reflinks when importing from the alternate storage into the ostree repo?

#20 links to containers/container-libs#144

jlebon · 2025-10-07T15:42:36Z

Do we get reflinks when importing from the alternate storage into the ostree repo?

#20 links to containers/container-libs#144

Hmm, that's about copying between two container storages, but I was asking about the ostree import. But #20 does say:

From there, we do a reflink/hardlink copy into the ostree store (i.e. This would duplicate metadata, but not data on disk. Reflinks should always work.

Which kinda answers the question.

Basically without reflinks into the ostree storage, IMO this makes it a lot less obvious that this new option is objectively better and it instead becomes a discussion of trade-offs.

jmarrero · 2025-12-02T20:06:50Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces experimental support for unified storage, allowing bootc to pull images into its own container storage before importing them. This is a significant feature that touches installation, upgrade, and switch paths. The changes include new CLI options, a new image set-unified command, and the core logic for the unified pull mechanism.

The implementation is comprehensive and includes new integration tests. My review has identified a few areas for improvement:

A bug in the image reference formatting for unified storage checks during installation.
Inconsistent error handling when checking for images in the unified store.
A potential panic due to an .unwrap() call on a Result.

Addressing these points will improve the robustness and maintainability of this new feature.

crates/lib/src/image.rs

crates/lib/src/install.rs

crates/lib/src/deploy.rs

cgwalters · 2025-12-02T20:51:58Z

On the progress status side maybe we can use https://lib.rs/crates/bollard to talk to podman and I think it should help us get data out of the events?

crates/lib/src/image.rs

jmarrero · 2025-12-10T17:33:10Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces experimental support for unified storage in bootc, a significant and well-implemented feature. The changes are comprehensive, adding the --experimental-unified-storage flag to install and switch commands, auto-detection for upgrade, and a new bootc image set-unified command for onboarding existing systems. The implementation correctly handles different sysroot paths and edge cases like localhost images. The inclusion of integration and TMT tests demonstrates a thorough approach to quality assurance. My review includes a few minor suggestions to improve code style and readability.

crates/lib/src/deploy.rs

jmarrero · 2025-12-10T17:44:27Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces experimental unified storage support for bootc install and related commands. The changes are extensive and well-thought-out, adding a new --experimental-unified-storage flag and the bootc image set-unified command. The implementation correctly handles different scenarios, such as install versus upgrade, and the special case of images originating from containers-storage. The addition of comprehensive integration and TMT tests is commendable. I've identified one high-severity logical issue in the implementation of bootc image set-unified where it could operate on a staged image instead of the booted one. I've also noted a correct and important bug fix in the image pruning logic. Overall, this is a solid contribution that adds a valuable feature.

crates/lib/src/image.rs

cgwalters

Thanks! A first pass review here, this is looking quite plausible!

crates/lib/src/cli.rs

cgwalters · 2025-12-10T19:18:03Z

crates/lib/src/deploy.rs

+pub(crate) async fn should_use_unified_storage(
+    store: &Storage,
+    imgref: &ImageReference,
+    explicit_flag: Option<bool>,


This seems strange to me; nothing AFAICS passes Some(false). And if Some(true) is passed the caller will just get back true. So why even call this?

crates/lib/src/image.rs

crates/lib/src/utils.rs

cgwalters · 2025-12-10T20:07:41Z

crates/lib/src/image.rs

+        // Check if the image already exists in the default containers-storage.
+        // This can happen if someone did a local build (e.g., podman build) and
+        // we don't want to overwrite it with an export from ostree.
+        ensure_floating_c_storage_initialized();


I think we can put this global init somewhere farther up?

crates/lib/src/image.rs

crates/system-reinstall-bootc/src/main.rs

crates/lib/src/image.rs

jmarrero · 2025-12-11T03:45:06Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces experimental support for unified storage in bootc install, which allows sharing container images with podman's storage to reduce disk usage and enable offline installations. The changes include a new --experimental-unified-storage flag, modifications to upgrade and switch to auto-detect and use unified storage, and special handling for localhost images. New integration and TMT tests are also added to cover these new code paths.

My review focuses on correctness and maintainability. I've identified a couple of areas for improvement:

An inconsistency in the image pulling strategy for unified storage, which could lead to unnecessary network operations.
A small opportunity to simplify code by removing a redundant check, improving readability and reflecting the guarantees provided by other functions.

Overall, the changes are well-structured and the addition of tests is great. Addressing these points will improve the robustness and maintainability of the new feature.

crates/lib/src/deploy.rs

crates/lib/src/image.rs

cgwalters · 2025-12-11T19:45:24Z

Hmm the failure of test-install is weird; I'm not seeing the job fail in e.g. #1846 or other PRs.

cgwalters · 2025-12-11T19:51:09Z

crates/lib/src/deploy.rs

+    // During install, this is the target disk's mount point; otherwise default to /sysroot
+    let sysroot_base = sysroot_path
+        .map(|p| p.to_string())
+        .unwrap_or_else(|| "/sysroot".to_string());


Let's clean this up post merge - should get this stuff from the Storage

cgwalters · 2025-12-11T19:56:15Z

crates/lib/src/install.rs

    // We must really be in a rootless container, or in some way
    // we're not part of the host user namespace.
-    ensure!(pid1_uid == 0, "{proc1} is owned by {pid1_uid}, not zero; this command must be run in the root user namespace (e.g. not rootless podman)");
+    ensure!(


Ideally we wouldn't reformat these - I have a suspicion that the test install failure is because we changed message output and that might have broken the expect() statements

crates/lib/src/install.rs

tmt/tests/booted/test-switch-to-unified.nu

crates/lib/src/install.rs

cgwalters

Marking as requested changes per incompatible change to install flow

Add an experimental --experimental-unified-storage flag to bootc install that uses bootc's container storage (/usr/lib/bootc/storage) to pull images first, then imports from there. This is the same approach used for logically bound images (LBIs). Background: The unified storage approach allows bootc to share container images with podman's storage, reducing disk space and enabling better integration with podman. Changes: - Add --experimental-unified-storage CLI flag to install subcommands - Add sysroot_path parameter to prepare_for_pull_unified() and pull_unified() to handle the different mount points during install vs upgrade/switch - Handle container-storage transport - Skip pull in prepare_for_pull_unified() if image already exists in bootc storage - Add TMT test for install with unified storage flag - Add TMT test for switching to unified storage on running system The sysroot_path fix is needed because during install the target disk is mounted at a specific path (e.g., /var/mnt), not /sysroot. Skopeo needs the actual filesystem path to find the bootc storage. Relates: bootc-dev#20 Assisted-by: Claude Code (Sonnet 4.5) Signed-off-by: Joseph Marrero Corchado <[email protected]>

jmarrero · 2025-12-12T12:50:32Z

Failures on rawhide are the same we saw in another PR yesterday: #1849

cgwalters

Cool, just two nits

cgwalters · 2025-12-12T13:20:05Z

docs/src/man/bootc-install-to-disk.8.md


    Verify the image can be fetched from the bootc image. Updates may fail when the installation host is authenticated with the registry but the pull secret is not in the bootc image

+**--experimental-unified-storage**


Hmm why did this show up in the man pages? Is our man page not handling hide = true? Or did you forget to regenerate? (We probably need a CI gate for this too)

cgwalters · 2025-12-12T13:22:01Z

crates/lib/src/spec.rs

+
+    /// Parse the transport string into a Transport enum.
+    pub fn transport(&self) -> Transport {
+        Transport::try_from(self.transport.as_str()).unwrap_or(Transport::Registry)


Under what scenarios would we want to silently fall back to registry if we fail to parse the input string? I think parsing should usually error out and not have silent fallbacks.

In the function just above this we do let transport = Transport::try_from(self.transport.as_str())?; which feels more correct to me.

(This shouldn't fail of course and per that TODO it'd be better to represent with a proper enum in this code too)

bootc-bot bot requested a review from cgwalters September 9, 2025 17:28

cgwalters reviewed Sep 9, 2025

View reviewed changes

jmarrero force-pushed the unified-storage-wip branch from 1baf1e8 to 9c3ef81 Compare October 9, 2025 18:15

jmarrero changed the title ~~WIP: Add --unified to use our container-storage for host images~~ WIP: Use our container-storage for host images if the refspec exists on it Oct 9, 2025

jmarrero changed the title ~~WIP: Use our container-storage for host images if the refspec exists on it~~ WIP: Use our container-storage for host images if the refspec exists Oct 9, 2025

jmarrero force-pushed the unified-storage-wip branch 2 times, most recently from e891e6c to be75055 Compare October 15, 2025 17:57

github-actions bot added the area/install Issues related to `bootc install` label Oct 15, 2025

jmarrero force-pushed the unified-storage-wip branch from 291fbd7 to fe10204 Compare November 11, 2025 19:53

jmarrero force-pushed the unified-storage-wip branch 3 times, most recently from 06be8fb to b801a17 Compare November 21, 2025 20:50

github-actions bot added the area/system-reinstall-bootc Issues related to system-reinstall-botoc label Nov 21, 2025

jmarrero force-pushed the unified-storage-wip branch from b801a17 to 68b1ba6 Compare November 21, 2025 20:56

jmarrero force-pushed the unified-storage-wip branch 3 times, most recently from ee300e3 to 50c4f70 Compare December 2, 2025 19:28

jmarrero changed the title ~~WIP: Use our container-storage for host images if the refspec exists~~ lib: Add experimental unified storage support for install Dec 2, 2025

gemini-code-assist bot reviewed Dec 2, 2025

View reviewed changes

crates/lib/src/image.rs Outdated Show resolved Hide resolved

crates/lib/src/install.rs Outdated Show resolved Hide resolved

crates/lib/src/deploy.rs Outdated Show resolved Hide resolved

cgwalters reviewed Dec 2, 2025

View reviewed changes

crates/lib/src/image.rs Outdated Show resolved Hide resolved

cgwalters reviewed Dec 4, 2025

View reviewed changes

crates/lib/src/image.rs Outdated Show resolved Hide resolved

jmarrero force-pushed the unified-storage-wip branch 4 times, most recently from 3803e4c to bb9883e Compare December 10, 2025 15:41

gemini-code-assist bot reviewed Dec 10, 2025

View reviewed changes

crates/lib/src/deploy.rs Show resolved Hide resolved

crates/lib/src/deploy.rs Show resolved Hide resolved

crates/lib/src/deploy.rs Outdated Show resolved Hide resolved

jmarrero force-pushed the unified-storage-wip branch from 04620ac to 7651032 Compare December 10, 2025 17:44

gemini-code-assist bot reviewed Dec 10, 2025

View reviewed changes

crates/lib/src/image.rs Outdated Show resolved Hide resolved

cgwalters requested changes Dec 10, 2025

View reviewed changes

jmarrero force-pushed the unified-storage-wip branch from 7651032 to ddbba51 Compare December 11, 2025 03:41

gemini-code-assist bot reviewed Dec 11, 2025

View reviewed changes

crates/lib/src/deploy.rs Show resolved Hide resolved

crates/lib/src/image.rs Show resolved Hide resolved

jmarrero force-pushed the unified-storage-wip branch 3 times, most recently from ba8e4ef to 7f11db8 Compare December 11, 2025 10:08

cgwalters mentioned this pull request Dec 11, 2025

Store container manifest #1830

Merged

cgwalters approved these changes Dec 11, 2025

View reviewed changes

cgwalters reviewed Dec 11, 2025

View reviewed changes

tmt/tests/booted/test-switch-to-unified.nu Show resolved Hide resolved

crates/lib/src/install.rs Outdated Show resolved Hide resolved

cgwalters requested changes Dec 11, 2025

View reviewed changes

jmarrero force-pushed the unified-storage-wip branch from 7f11db8 to a3aea8e Compare December 12, 2025 03:25

github-actions bot added the area/documentation Updates to the documentation label Dec 12, 2025

jmarrero force-pushed the unified-storage-wip branch 2 times, most recently from 013df9f to 62d48ab Compare December 12, 2025 04:27

jmarrero force-pushed the unified-storage-wip branch from 62d48ab to 52f5e25 Compare December 12, 2025 04:53

jmarrero mentioned this pull request Dec 12, 2025

rawhide tests "failing with error: Status: loading sysroot: Unexpected state: /run/ostree-booted found, but no /boot/loader directory" #1849

Open

jmarrero requested a review from cgwalters December 12, 2025 12:48

cgwalters approved these changes Dec 12, 2025

View reviewed changes

cgwalters mentioned this pull request Dec 12, 2025

Fix signature handling with additionalimagestore bootc-dev/bcvk#166

Open


		Verify the image can be fetched from the bootc image. Updates may fail when the installation host is authenticated with the registry but the pull secret is not in the bootc image

		--experimental-unified-storage

lib: Add experimental unified storage support for install #1601

Are you sure you want to change the base?

lib: Add experimental unified storage support for install #1601

Uh oh!

Conversation

jmarrero commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jlebon commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Oct 7, 2025

Uh oh!

jlebon commented Oct 7, 2025

Uh oh!

jmarrero commented Dec 2, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgwalters commented Dec 2, 2025

Uh oh!

Uh oh!

Uh oh!

jmarrero commented Dec 10, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmarrero commented Dec 10, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmarrero commented Dec 11, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

jmarrero commented Sep 9, 2025 •

edited

Loading

jlebon commented Oct 7, 2025 •

edited

Loading